[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:05:50 dist-test-slave-hmbw krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:05:50 dist-test-slave-hmbw krb5kdc[395](info): set up 2 sockets
Sep 26 03:05:50 dist-test-slave-hmbw krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 03:05:51.120083 388 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.001s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:05:51 dist-test-slave-hmbw krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:05:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:05:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:05:51.294690 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34921
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:05:51.633972 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:05:51.634320 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:05:51.634409 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:05:51.641717 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:05:51.641840 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:05:51.641886 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:05:51.641940 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:05:51.641983 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:05:51.652434 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:05:51.653759 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:05:51.655176 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:05:51.668363 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:51.668402 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:51.668443 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:05:51.669462 408 server_base.cc:1034] running on GCE node
I20260926 03:05:51.670362 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:05:51.672086 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:05:51.673352 408 hybrid_clock.cc:648] HybridClock initialized: now 1790391951673253 us; error 96 us; skew 500 ppm
I20260926 03:05:51.673749 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:05:51 dist-test-slave-hmbw krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:05:51.677352 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:05:51.679028 408 webserver.cc:460] Webserver started at http://127.0.97.62:43649/ using document root <none> and password file <none>
I20260926 03:05:51.679778 408 fs_manager.cc:359] Metadata directory not provided
I20260926 03:05:51.679898 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:05:51.680200 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:05:51.683444 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "297a58bef4874ce8a3c4d3dc8abb8fd5"
format_stamp: "Formatted at 2026-09-26 03:05:51 on dist-test-slave-hmbw"
server_key: "569fbd64270a66870b0491573337a179"
server_key_iv: "c3eec12dc96f327c55aee5cf383ef232"
server_key_version: "encryptionkey@0"
I20260926 03:05:51.684243 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "297a58bef4874ce8a3c4d3dc8abb8fd5"
format_stamp: "Formatted at 2026-09-26 03:05:51 on dist-test-slave-hmbw"
server_key: "569fbd64270a66870b0491573337a179"
server_key_iv: "c3eec12dc96f327c55aee5cf383ef232"
server_key_version: "encryptionkey@0"
I20260926 03:05:51.690102 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 03:05:51.694082 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:05:51.695660 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:05:51.695861 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "297a58bef4874ce8a3c4d3dc8abb8fd5"
format_stamp: "Formatted at 2026-09-26 03:05:51 on dist-test-slave-hmbw"
server_key: "569fbd64270a66870b0491573337a179"
server_key_iv: "c3eec12dc96f327c55aee5cf383ef232"
server_key_version: "encryptionkey@0"
I20260926 03:05:51.696062 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:05:51.713883 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:05:51.718338 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:05:51.718657 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:05:51.738763 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34921
I20260926 03:05:51.738783 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34921 every 8 connection(s)
I20260926 03:05:51.740195 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:05:51.741776 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 408
I20260926 03:05:51.742034 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:05:51.743965 388 external_mini_cluster.cc:1428] Setting key 7cb5974e0d204cad212ebb7d191d8b53
I20260926 03:05:51.744464 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:05:51.759650 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 297a58bef4874ce8a3c4d3dc8abb8fd5: Bootstrap starting.
I20260926 03:05:51.765230 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 297a58bef4874ce8a3c4d3dc8abb8fd5: Neither blocks nor log segments found. Creating new log.
I20260926 03:05:51.766425 475 log.cc:824] T 00000000000000000000000000000000 P 297a58bef4874ce8a3c4d3dc8abb8fd5: Log is configured to *not* fsync() on all Append() calls
I20260926 03:05:51.769508 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 297a58bef4874ce8a3c4d3dc8abb8fd5: No bootstrap required, opened a new log
Sep 26 03:05:51 dist-test-slave-hmbw krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:05:51.780922 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 297a58bef4874ce8a3c4d3dc8abb8fd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "297a58bef4874ce8a3c4d3dc8abb8fd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34921 } }
I20260926 03:05:51.781350 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 297a58bef4874ce8a3c4d3dc8abb8fd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:05:51.781436 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 297a58bef4874ce8a3c4d3dc8abb8fd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297a58bef4874ce8a3c4d3dc8abb8fd5, State: Initialized, Role: FOLLOWER
I20260926 03:05:51.782260 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 297a58bef4874ce8a3c4d3dc8abb8fd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "297a58bef4874ce8a3c4d3dc8abb8fd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34921 } }
I20260926 03:05:51.782536 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 297a58bef4874ce8a3c4d3dc8abb8fd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:05:51.782685 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 297a58bef4874ce8a3c4d3dc8abb8fd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:05:51.782871 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 297a58bef4874ce8a3c4d3dc8abb8fd5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:05:51.786542 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 297a58bef4874ce8a3c4d3dc8abb8fd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "297a58bef4874ce8a3c4d3dc8abb8fd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34921 } }
I20260926 03:05:51.787384 475 leader_election.cc:302] T 00000000000000000000000000000000 P 297a58bef4874ce8a3c4d3dc8abb8fd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 297a58bef4874ce8a3c4d3dc8abb8fd5; no voters:
I20260926 03:05:51.788146 475 leader_election.cc:288] T 00000000000000000000000000000000 P 297a58bef4874ce8a3c4d3dc8abb8fd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:05:51.788300 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 297a58bef4874ce8a3c4d3dc8abb8fd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:05:51.789996 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 297a58bef4874ce8a3c4d3dc8abb8fd5 [term 1 LEADER]: Becoming Leader. State: Replica: 297a58bef4874ce8a3c4d3dc8abb8fd5, State: Running, Role: LEADER
I20260926 03:05:51.790566 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 297a58bef4874ce8a3c4d3dc8abb8fd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "297a58bef4874ce8a3c4d3dc8abb8fd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34921 } }
I20260926 03:05:51.790824 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 297a58bef4874ce8a3c4d3dc8abb8fd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:05:51.793529 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:51.748039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52848 (local address 127.0.97.62:34921)
0926 03:05:51.748887 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:51.748940 (+ 53us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:51.750594 (+ 1654us) server_negotiation.cc:407] Connection header received
0926 03:05:51.764408 (+ 13814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:51.764441 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:51.764979 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:51.765477 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:51.768865 (+ 3388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:51.770292 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:51.772237 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:51.773216 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:51.773522 (+ 306us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:51.781968 (+ 8446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:51.781978 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:51.782000 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:51.782057 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:51.786192 (+ 4135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:51.787413 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:51.787425 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:51.787435 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:51.787691 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:51.788564 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:51.788574 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:51.788578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:51.789874 (+ 1296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:51.791024 (+ 1150us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:51.791465 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 03:05:51.791917 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":361,"threads_started":1}
I20260926 03:05:51.797169 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 297a58bef4874ce8a3c4d3dc8abb8fd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "297a58bef4874ce8a3c4d3dc8abb8fd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "297a58bef4874ce8a3c4d3dc8abb8fd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34921 } } }
I20260926 03:05:51.797606 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 297a58bef4874ce8a3c4d3dc8abb8fd5 [sys.catalog]: This master's current role is: LEADER
I20260926 03:05:51.798066 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 297a58bef4874ce8a3c4d3dc8abb8fd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 297a58bef4874ce8a3c4d3dc8abb8fd5. Latest consensus state: current_term: 1 leader_uuid: "297a58bef4874ce8a3c4d3dc8abb8fd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "297a58bef4874ce8a3c4d3dc8abb8fd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34921 } } }
I20260926 03:05:51.798357 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 297a58bef4874ce8a3c4d3dc8abb8fd5 [sys.catalog]: This master's current role is: LEADER
I20260926 03:05:51.807336 487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:05:51.816267 487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:05:51.828097 487 catalog_manager.cc:1347] Generated new cluster ID: 5a10671123994e0f8313985bb4f43bcb
I20260926 03:05:51.828222 487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:05:51.843179 487 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:05:51.844521 487 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:05:51.850087 487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 297a58bef4874ce8a3c4d3dc8abb8fd5: Generated new TSK 0
I20260926 03:05:51.851096 487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:05:51.899009 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34921
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:05:52.275728 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:05:52.276026 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:05:52.276134 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:05:52.283722 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:05:52.283901 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:05:52.284448 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:05:52.295094 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34921
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:05:52.296523 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:05:52.297932 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:05:52.315505 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:52.315613 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:52.315546 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:05:52.317306 501 server_base.cc:1034] running on GCE node
I20260926 03:05:52.318064 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:05:52.319106 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:05:52.320326 501 hybrid_clock.cc:648] HybridClock initialized: now 1790391952320275 us; error 42 us; skew 500 ppm
I20260926 03:05:52.320644 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:05:52 dist-test-slave-hmbw krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:05:52.323971 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:05:52.325599 501 webserver.cc:460] Webserver started at http://127.0.97.1:40931/ using document root <none> and password file <none>
I20260926 03:05:52.326314 501 fs_manager.cc:359] Metadata directory not provided
I20260926 03:05:52.326432 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:05:52.326763 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:05:52.329769 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-0/data/instance:
uuid: "3eb7209f6b0940a7a94c6f3327c5d0e9"
format_stamp: "Formatted at 2026-09-26 03:05:52 on dist-test-slave-hmbw"
server_key: "ed92efe4ab9479f0cdb149944f106cfb"
server_key_iv: "02817fa240f1db6a00f9452e136e9611"
server_key_version: "encryptionkey@0"
I20260926 03:05:52.330653 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3eb7209f6b0940a7a94c6f3327c5d0e9"
format_stamp: "Formatted at 2026-09-26 03:05:52 on dist-test-slave-hmbw"
server_key: "ed92efe4ab9479f0cdb149944f106cfb"
server_key_iv: "02817fa240f1db6a00f9452e136e9611"
server_key_version: "encryptionkey@0"
I20260926 03:05:52.336606 501 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 03:05:52.340426 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:05:52.341952 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:05:52.342147 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "3eb7209f6b0940a7a94c6f3327c5d0e9"
format_stamp: "Formatted at 2026-09-26 03:05:52 on dist-test-slave-hmbw"
server_key: "ed92efe4ab9479f0cdb149944f106cfb"
server_key_iv: "02817fa240f1db6a00f9452e136e9611"
server_key_version: "encryptionkey@0"
I20260926 03:05:52.342345 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:05:52.368106 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:05:52.373026 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:05:52.373373 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:05:52.374553 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:05:52.376036 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:05:52.376119 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:05:52.376313 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:05:52.376417 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:05:52.413956 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43495
I20260926 03:05:52.414000 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43495 every 8 connection(s)
I20260926 03:05:52.415316 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:05:52.420197 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 501
I20260926 03:05:52.420392 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:05:52.420966 388 external_mini_cluster.cc:1428] Setting key c7b8c5ce81be53dae79b63be653a46d1
Sep 26 03:05:52 dist-test-slave-hmbw krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:05:52.438771 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:52.418021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57579 (local address 127.0.97.62:34921)
0926 03:05:52.418451 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:52.418456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:52.419587 (+ 1131us) server_negotiation.cc:407] Connection header received
0926 03:05:52.422227 (+ 2640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:52.422233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:52.422316 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:52.422441 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:52.424282 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:52.425355 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:52.427257 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:52.427833 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:52.427938 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:52.432425 (+ 4487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:52.432433 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:52.432439 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:52.432488 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:52.435368 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:52.436223 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:52.436228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:52.436230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:52.436356 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:52.436809 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:52.436813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:52.436815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:52.437103 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:52.437256 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:52.438075 (+ 819us) server_negotiation.cc:299] Negotiation successful
0926 03:05:52.438322 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":219,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 03:05:52.439793 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:52.418586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34921 (local address 127.0.97.1:57579)
0926 03:05:52.419401 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:52.419444 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:52.421775 (+ 2331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:52.422795 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:52.422806 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:52.423447 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:05:52.424074 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:52.424093 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:52.425569 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:52.425576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:52.427087 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:52.427104 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:52.428102 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:52.428120 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:52.428455 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:52.429389 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:05:52.429416 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:05:52.432089 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:05:52.435561 (+ 3472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:52.435571 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:52.435596 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:52.436076 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:52.436467 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:52.436472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:52.436475 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:52.436678 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:52.437248 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:05:52.437257 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:05:52.437643 (+ 386us) client_negotiation.cc:769] Sending connection context
0926 03:05:52.437898 (+ 255us) client_negotiation.cc:240] Negotiation successful
0926 03:05:52.438472 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":330,"threads_started":1}
I20260926 03:05:52.441609 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34921
I20260926 03:05:52.441967 629 heartbeater.cc:464] Registering TS with master...
I20260926 03:05:52.442847 629 heartbeater.cc:511] Master 127.0.97.62:34921 requested a full tablet report, sending...
I20260926 03:05:52.445335 440 ts_manager.cc:194] Registered new tserver with Master: 3eb7209f6b0940a7a94c6f3327c5d0e9 (127.0.97.1:43495)
I20260926 03:05:52.447383 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:05:52.466403 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34921
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:05:52.813696 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:05:52.813987 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:05:52.814080 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:05:52.821547 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:05:52.821673 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:05:52.821796 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:05:52.832203 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34921
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:05:52.833482 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:05:52.834941 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:05:52.849525 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:52.849579 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:52.849869 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:05:52.852376 635 server_base.cc:1034] running on GCE node
I20260926 03:05:52.853065 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:05:52.854190 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:05:52.855484 635 hybrid_clock.cc:648] HybridClock initialized: now 1790391952855428 us; error 56 us; skew 500 ppm
I20260926 03:05:52.855829 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:05:52 dist-test-slave-hmbw krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:05:52.859165 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:05:52.860778 635 webserver.cc:460] Webserver started at http://127.0.97.2:44759/ using document root <none> and password file <none>
I20260926 03:05:52.861493 635 fs_manager.cc:359] Metadata directory not provided
I20260926 03:05:52.861610 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:05:52.861923 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:05:52.864929 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-1/data/instance:
uuid: "7162654046e44057aed701f1e93a128b"
format_stamp: "Formatted at 2026-09-26 03:05:52 on dist-test-slave-hmbw"
server_key: "83f076632fe577eb54a9e6cfdcb3c027"
server_key_iv: "0e624550a1cca03738519ed07a49e1f4"
server_key_version: "encryptionkey@0"
I20260926 03:05:52.865746 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7162654046e44057aed701f1e93a128b"
format_stamp: "Formatted at 2026-09-26 03:05:52 on dist-test-slave-hmbw"
server_key: "83f076632fe577eb54a9e6cfdcb3c027"
server_key_iv: "0e624550a1cca03738519ed07a49e1f4"
server_key_version: "encryptionkey@0"
I20260926 03:05:52.872031 635 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20260926 03:05:52.875862 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:05:52.877389 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:05:52.877952 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "7162654046e44057aed701f1e93a128b"
format_stamp: "Formatted at 2026-09-26 03:05:52 on dist-test-slave-hmbw"
server_key: "83f076632fe577eb54a9e6cfdcb3c027"
server_key_iv: "0e624550a1cca03738519ed07a49e1f4"
server_key_version: "encryptionkey@0"
I20260926 03:05:52.878165 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:05:52.892181 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:05:52.896341 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:05:52.896613 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:05:52.897593 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:05:52.898976 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:05:52.899055 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:05:52.899171 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:05:52.899226 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:05:52.933770 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43183
I20260926 03:05:52.933815 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43183 every 8 connection(s)
I20260926 03:05:52.935082 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:05:52.943491 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 635
I20260926 03:05:52.943658 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:05:52.944053 388 external_mini_cluster.cc:1428] Setting key a9da5c4905cf5dc17e83cce5f699ea0d
Sep 26 03:05:52 dist-test-slave-hmbw krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:05:52.956174 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:52.937525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50801 (local address 127.0.97.62:34921)
0926 03:05:52.937741 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:52.937745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:52.938898 (+ 1153us) server_negotiation.cc:407] Connection header received
0926 03:05:52.940797 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:52.940802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:52.940890 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:52.940973 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:52.942531 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:52.943350 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:52.945264 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:52.945752 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:52.945845 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:52.949851 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:52.949858 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:52.949862 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:52.949900 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:52.952887 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:52.953717 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:52.953722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:52.953724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:52.953874 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:52.954306 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:52.954311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:52.954314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:52.954620 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:52.954770 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:52.955709 (+ 939us) server_negotiation.cc:299] Negotiation successful
0926 03:05:52.955921 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:05:52.957815 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:52.937981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34921 (local address 127.0.97.2:50801)
0926 03:05:52.938741 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:52.938786 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:52.940519 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:52.941239 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:52.941248 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:52.941762 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:05:52.942343 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:52.942359 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:52.943546 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:52.943550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:52.945127 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:52.945142 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:52.945986 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:52.945994 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:52.946232 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:52.946969 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:05:52.947002 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:05:52.949596 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:05:52.953058 (+ 3462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:52.953065 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:52.953080 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:52.953589 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:52.953982 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:52.953986 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:52.953988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:52.954195 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:52.954770 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:05:52.954780 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:05:52.955285 (+ 505us) client_negotiation.cc:769] Sending connection context
0926 03:05:52.955630 (+ 345us) client_negotiation.cc:240] Negotiation successful
0926 03:05:52.956255 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":304,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 03:05:52.959477 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34921
I20260926 03:05:52.959793 763 heartbeater.cc:464] Registering TS with master...
I20260926 03:05:52.960598 763 heartbeater.cc:511] Master 127.0.97.62:34921 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 03:05:52.962446 440 ts_manager.cc:194] Registered new tserver with Master: 7162654046e44057aed701f1e93a128b (127.0.97.2:43183)
I20260926 03:05:52.963294 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:05:52.985078 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34921
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:05:53.332479 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:05:53.332760 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:05:53.332860 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:05:53.340162 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:05:53.340322 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:05:53.340468 768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:05:53.351361 768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34921
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:05:53.352613 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:05:53.353896 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:05:53.367738 773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:53.367713 774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:53.368039 776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:05:53.368568 768 server_base.cc:1034] running on GCE node
I20260926 03:05:53.369163 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:05:53.370131 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:05:53.371379 768 hybrid_clock.cc:648] HybridClock initialized: now 1790391953371337 us; error 37 us; skew 500 ppm
I20260926 03:05:53.371665 768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:05:53 dist-test-slave-hmbw krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:05:53.374933 768 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:05:53.376591 768 webserver.cc:460] Webserver started at http://127.0.97.3:35135/ using document root <none> and password file <none>
I20260926 03:05:53.377247 768 fs_manager.cc:359] Metadata directory not provided
I20260926 03:05:53.377353 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:05:53.377605 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:05:53.380578 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-2/data/instance:
uuid: "ecb4b6e672aa4dda93f47456081d2dcd"
format_stamp: "Formatted at 2026-09-26 03:05:53 on dist-test-slave-hmbw"
server_key: "18279d486bb38f1e3b296c9f4aa0d2f3"
server_key_iv: "460c745a52a0ab5da6a9903f189d8f24"
server_key_version: "encryptionkey@0"
I20260926 03:05:53.381387 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ecb4b6e672aa4dda93f47456081d2dcd"
format_stamp: "Formatted at 2026-09-26 03:05:53 on dist-test-slave-hmbw"
server_key: "18279d486bb38f1e3b296c9f4aa0d2f3"
server_key_iv: "460c745a52a0ab5da6a9903f189d8f24"
server_key_version: "encryptionkey@0"
I20260926 03:05:53.387382 768 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260926 03:05:53.391510 783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:05:53.393098 768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:05:53.393272 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "ecb4b6e672aa4dda93f47456081d2dcd"
format_stamp: "Formatted at 2026-09-26 03:05:53 on dist-test-slave-hmbw"
server_key: "18279d486bb38f1e3b296c9f4aa0d2f3"
server_key_iv: "460c745a52a0ab5da6a9903f189d8f24"
server_key_version: "encryptionkey@0"
I20260926 03:05:53.393481 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:05:53.417667 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:05:53.422640 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:05:53.422876 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:05:53.423779 768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:05:53.425076 768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:05:53.425150 768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:05:53.425231 768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:05:53.425275 768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:05:53.450685 629 heartbeater.cc:503] Master 127.0.97.62:34921 was elected leader, sending a full tablet report...
I20260926 03:05:53.459550 768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34729
I20260926 03:05:53.459601 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34729 every 8 connection(s)
I20260926 03:05:53.461002 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:05:53.462795 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 768
I20260926 03:05:53.462978 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391950058021-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:05:53.463409 388 external_mini_cluster.cc:1428] Setting key 320db7624199a534110346b5608af8d9
Sep 26 03:05:53 dist-test-slave-hmbw krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:05:53.486143 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:53.464194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57087 (local address 127.0.97.62:34921)
0926 03:05:53.470953 (+ 6759us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:53.470957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:53.470977 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:05:53.471028 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:53.471031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:53.471095 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:53.471206 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:53.472955 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:53.473732 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:53.475928 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:53.476333 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:53.476446 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:53.480236 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:53.480243 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:53.480247 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:53.480354 (+ 107us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:53.483076 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:53.483881 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:53.483886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:53.483888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:53.484004 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:53.484469 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:53.484474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:53.484476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:53.484748 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:53.484899 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:53.485661 (+ 762us) server_negotiation.cc:299] Negotiation successful
0926 03:05:53.485874 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6640,"thread_start_us":6504,"threads_started":1}
I20260926 03:05:53.487174 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:53.464185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34921 (local address 127.0.97.3:57087)
0926 03:05:53.465009 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:53.465065 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:53.467010 (+ 1945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:53.471568 (+ 4558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:53.471579 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:53.472139 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:05:53.472760 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:53.472790 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:53.473926 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:53.473930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:53.475766 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:53.475782 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:53.476570 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:53.476577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:53.476816 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:53.477474 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:05:53.477497 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:05:53.480025 (+ 2528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:05:53.483240 (+ 3215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:53.483248 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:53.483262 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:53.483743 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:53.484125 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:53.484129 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:53.484132 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:53.484350 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:53.484927 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:05:53.484934 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:05:53.485307 (+ 373us) client_negotiation.cc:769] Sending connection context
0926 03:05:53.485548 (+ 241us) client_negotiation.cc:240] Negotiation successful
0926 03:05:53.485977 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":346,"threads_started":1}
I20260926 03:05:53.488636 896 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34921
I20260926 03:05:53.488968 896 heartbeater.cc:464] Registering TS with master...
I20260926 03:05:53.489749 896 heartbeater.cc:511] Master 127.0.97.62:34921 requested a full tablet report, sending...
I20260926 03:05:53.491458 440 ts_manager.cc:194] Registered new tserver with Master: ecb4b6e672aa4dda93f47456081d2dcd (127.0.97.3:34729)
I20260926 03:05:53.492441 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57087
I20260926 03:05:53.492749 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 03:05:53 dist-test-slave-hmbw krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391953, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 03:05:53 dist-test-slave-hmbw krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391953, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:05:53.536301 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:53.520418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57428 (local address 127.0.97.1:43495)
0926 03:05:53.521206 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:53.521216 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:53.521516 (+ 300us) server_negotiation.cc:407] Connection header received
0926 03:05:53.521689 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:53.521696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:53.521918 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:53.522070 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:53.523565 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:53.524581 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:53.525847 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:53.526411 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:53.526553 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:53.530460 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:53.530473 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:53.530488 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:53.530562 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:53.533478 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:53.534155 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:53.534163 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:53.534166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:53.534295 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:53.534718 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:53.534723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:53.534726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:53.535073 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:53.535348 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:53.535722 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 03:05:53.535980 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":412,"threads_started":1}
W20260926 03:05:53.537631 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57428
Sep 26 03:05:53 dist-test-slave-hmbw krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 03:05:53 dist-test-slave-hmbw krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:05:53.582783 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:53.566810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57440 (local address 127.0.97.1:43495)
0926 03:05:53.567908 (+ 1098us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:53.567915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:53.567940 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:05:53.568007 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:53.568011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:53.568113 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:53.568217 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:53.569628 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:53.570309 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:53.571777 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:53.572239 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:53.572362 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:53.576961 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:53.576966 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:53.576970 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:53.577011 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:53.579981 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:53.580757 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:53.580762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:53.580764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:53.580958 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:53.581460 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:53.581464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:53.581466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:53.581729 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:53.581943 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:53.582300 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 03:05:53.582526 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333}
I20260926 03:05:53.584524 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 501
I20260926 03:05:53.598937 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:53.585569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57450 (local address 127.0.97.1:43495)
0926 03:05:53.585829 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:53.585835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:53.585991 (+ 156us) server_negotiation.cc:407] Connection header received
0926 03:05:53.586158 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:53.586163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:53.586236 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:53.586319 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:53.587537 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:53.588236 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:53.589464 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:53.589984 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:53.590099 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:53.593439 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:53.593444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:53.593447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:53.593485 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:53.596184 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:53.596994 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:53.596999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:53.597002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:53.597149 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:53.597497 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:53.597501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:53.597503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:53.597823 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:53.598045 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:53.598351 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 03:05:53.598583 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260926 03:05:53.600047 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:05:53.683633 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 501
I20260926 03:05:53.697634 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 635
Sep 26 03:05:53 dist-test-slave-hmbw krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:05:53.714198 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:53.698450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44988 (local address 127.0.97.2:43183)
0926 03:05:53.699004 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:53.699011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:53.699038 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:05:53.699098 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:53.699105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:53.699315 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:53.699472 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:53.700651 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:53.701781 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:53.703341 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:53.703980 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:53.704095 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:53.708445 (+ 4350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:53.708462 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:53.708479 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:53.708521 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:53.711478 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:53.712196 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:53.712204 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:53.712206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:53.712388 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:53.712770 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:53.712774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:53.712776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:53.713112 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:53.713382 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:53.713687 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 03:05:53.713895 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":275,"threads_started":1}
I20260926 03:05:53.715526 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:05:53.796226 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 635
I20260926 03:05:53.815466 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 768
Sep 26 03:05:53 dist-test-slave-hmbw krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:05:53.831462 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:53.816323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50172 (local address 127.0.97.3:34729)
0926 03:05:53.816879 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:53.816886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:53.816913 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:05:53.816973 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:53.816979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:53.817190 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:53.817358 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:53.818606 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:53.819631 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:53.820897 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:53.821469 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:53.821583 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:53.825791 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:53.825807 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:53.825824 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:53.825861 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:53.828695 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:53.829389 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:53.829398 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:53.829400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:53.829521 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:53.829901 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:53.829905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:53.829907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:53.830262 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:53.830562 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:53.830864 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 03:05:53.831120 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":262,"threads_started":1}
I20260926 03:05:53.832783 891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:05:53.916062 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 768
I20260926 03:05:53.929209 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 408
I20260926 03:05:53.930406 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:05:54.031347 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 408
2026-09-26T03:05:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3930 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:05:54 dist-test-slave-hmbw krb5kdc[918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:05:54 dist-test-slave-hmbw krb5kdc[918](info): set up 2 sockets
Sep 26 03:05:54 dist-test-slave-hmbw krb5kdc[918](info): commencing operation
krb5kdc: starting...
W20260926 03:05:55.386353 388 mini_kdc.cc:121] Time spent starting KDC: real 1.315s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:05:55 dist-test-slave-hmbw krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:05:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:05:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:05:55.563685 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39191
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39191
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:05:55.885327 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:05:55.885649 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:05:55.885767 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:05:55.893769 931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:05:55.893934 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:05:55.893996 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:05:55.894078 931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:05:55.894131 931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:05:55.905470 931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:05:55.907014 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:05:55.908413 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:05:55.920691 936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:55.920730 937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:55.920660 939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:05:55.921586 931 server_base.cc:1034] running on GCE node
I20260926 03:05:55.922394 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:05:55.923928 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:05:55.925179 931 hybrid_clock.cc:648] HybridClock initialized: now 1790391955925115 us; error 44 us; skew 500 ppm
I20260926 03:05:55.925575 931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:05:55 dist-test-slave-hmbw krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:05:55.928949 931 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:05:55.930684 931 webserver.cc:460] Webserver started at http://127.0.97.62:39029/ using document root <none> and password file <none>
I20260926 03:05:55.931439 931 fs_manager.cc:359] Metadata directory not provided
I20260926 03:05:55.931547 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:05:55.931872 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:05:55.935148 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "7940365c50c444a9a90eb618d39aca0b"
format_stamp: "Formatted at 2026-09-26 03:05:55 on dist-test-slave-hmbw"
server_key: "e127f2cc21b920366b689875798bf8bc"
server_key_iv: "af1556ad9256e08fe8a923553c7a7b5d"
server_key_version: "encryptionkey@0"
I20260926 03:05:55.935971 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "7940365c50c444a9a90eb618d39aca0b"
format_stamp: "Formatted at 2026-09-26 03:05:55 on dist-test-slave-hmbw"
server_key: "e127f2cc21b920366b689875798bf8bc"
server_key_iv: "af1556ad9256e08fe8a923553c7a7b5d"
server_key_version: "encryptionkey@0"
I20260926 03:05:55.941359 931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:05:55.945123 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:05:55.946713 931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:05:55.946877 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "7940365c50c444a9a90eb618d39aca0b"
format_stamp: "Formatted at 2026-09-26 03:05:55 on dist-test-slave-hmbw"
server_key: "e127f2cc21b920366b689875798bf8bc"
server_key_iv: "af1556ad9256e08fe8a923553c7a7b5d"
server_key_version: "encryptionkey@0"
I20260926 03:05:55.947052 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:05:55.970966 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:05:55.975482 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:05:55.975718 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:05:55.993361 931 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39191
I20260926 03:05:55.993383 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39191 every 8 connection(s)
I20260926 03:05:55.994936 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:05:55.998984 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:05:56.001055 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 931
I20260926 03:05:56.001245 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:05:56.001905 388 external_mini_cluster.cc:1428] Setting key cb0dd8e60b930a1c4142b25f53a1d296
I20260926 03:05:56.011404 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7940365c50c444a9a90eb618d39aca0b: Bootstrap starting.
I20260926 03:05:56.015929 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7940365c50c444a9a90eb618d39aca0b: Neither blocks nor log segments found. Creating new log.
I20260926 03:05:56.017423 998 log.cc:824] T 00000000000000000000000000000000 P 7940365c50c444a9a90eb618d39aca0b: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:05:56 dist-test-slave-hmbw krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:05:56.021374 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7940365c50c444a9a90eb618d39aca0b: No bootstrap required, opened a new log
I20260926 03:05:56.028873 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:56.004003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60232 (local address 127.0.97.62:39191)
0926 03:05:56.004814 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:56.004827 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:56.004869 (+ 42us) server_negotiation.cc:407] Connection header received
0926 03:05:56.006601 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:56.006626 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:56.007057 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:56.007425 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:56.009224 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:56.010894 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:56.012676 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:56.013445 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:56.013594 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:56.019222 (+ 5628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:56.019233 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:56.019256 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:56.019308 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:56.023774 (+ 4466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:56.024625 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:56.024635 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:56.024647 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:56.024873 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:56.025263 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:56.025268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:56.025271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:56.026029 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:56.026308 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:56.026834 (+ 526us) server_negotiation.cc:299] Negotiation successful
0926 03:05:56.027357 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":373,"threads_started":1}
I20260926 03:05:56.031648 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7940365c50c444a9a90eb618d39aca0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7940365c50c444a9a90eb618d39aca0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39191 } }
I20260926 03:05:56.032032 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7940365c50c444a9a90eb618d39aca0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:05:56.032145 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7940365c50c444a9a90eb618d39aca0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7940365c50c444a9a90eb618d39aca0b, State: Initialized, Role: FOLLOWER
I20260926 03:05:56.032796 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7940365c50c444a9a90eb618d39aca0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7940365c50c444a9a90eb618d39aca0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39191 } }
I20260926 03:05:56.033028 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7940365c50c444a9a90eb618d39aca0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:05:56.033157 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7940365c50c444a9a90eb618d39aca0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:05:56.033306 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7940365c50c444a9a90eb618d39aca0b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:05:56.035598 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7940365c50c444a9a90eb618d39aca0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7940365c50c444a9a90eb618d39aca0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39191 } }
I20260926 03:05:56.036235 998 leader_election.cc:302] T 00000000000000000000000000000000 P 7940365c50c444a9a90eb618d39aca0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7940365c50c444a9a90eb618d39aca0b; no voters:
I20260926 03:05:56.036940 998 leader_election.cc:288] T 00000000000000000000000000000000 P 7940365c50c444a9a90eb618d39aca0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:05:56.037071 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7940365c50c444a9a90eb618d39aca0b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:05:56.037678 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7940365c50c444a9a90eb618d39aca0b [term 1 LEADER]: Becoming Leader. State: Replica: 7940365c50c444a9a90eb618d39aca0b, State: Running, Role: LEADER
I20260926 03:05:56.038268 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7940365c50c444a9a90eb618d39aca0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7940365c50c444a9a90eb618d39aca0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39191 } }
I20260926 03:05:56.038970 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7940365c50c444a9a90eb618d39aca0b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:05:56.041148 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7940365c50c444a9a90eb618d39aca0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7940365c50c444a9a90eb618d39aca0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7940365c50c444a9a90eb618d39aca0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39191 } } }
I20260926 03:05:56.041662 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7940365c50c444a9a90eb618d39aca0b [sys.catalog]: This master's current role is: LEADER
I20260926 03:05:56.041169 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7940365c50c444a9a90eb618d39aca0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7940365c50c444a9a90eb618d39aca0b. Latest consensus state: current_term: 1 leader_uuid: "7940365c50c444a9a90eb618d39aca0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7940365c50c444a9a90eb618d39aca0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39191 } } }
I20260926 03:05:56.041790 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7940365c50c444a9a90eb618d39aca0b [sys.catalog]: This master's current role is: LEADER
I20260926 03:05:56.044384 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:05:56.050721 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:05:56.060490 1010 catalog_manager.cc:1347] Generated new cluster ID: 35f8d6917d134abc82810cb82c9d2333
I20260926 03:05:56.060609 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:05:56.078034 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:05:56.079507 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:05:56.085760 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7940365c50c444a9a90eb618d39aca0b: Generated new TSK 0
I20260926 03:05:56.086984 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:05:56.142935 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39191
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:05:56.530304 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:05:56.530685 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:05:56.530797 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:05:56.538308 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:05:56.538447 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:05:56.538610 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:05:56.548882 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39191
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:05:56.550185 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:05:56.551720 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:05:56.565327 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:56.565399 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:56.565380 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:05:56.566637 1024 server_base.cc:1034] running on GCE node
I20260926 03:05:56.567299 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:05:56.568490 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:05:56.569727 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790391956569670 us; error 56 us; skew 500 ppm
I20260926 03:05:56.570070 1024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:05:56 dist-test-slave-hmbw krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:05:56.573419 1024 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:05:56.575110 1024 webserver.cc:460] Webserver started at http://127.0.97.1:40089/ using document root <none> and password file <none>
I20260926 03:05:56.575956 1024 fs_manager.cc:359] Metadata directory not provided
I20260926 03:05:56.576084 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:05:56.576390 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:05:56.580842 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-0/data/instance:
uuid: "6fee0d8a1dd14538bbf905a157dfad21"
format_stamp: "Formatted at 2026-09-26 03:05:56 on dist-test-slave-hmbw"
server_key: "8f3482106d9074dec8a78b653f574079"
server_key_iv: "0a94918f294832208ca287d219f2a971"
server_key_version: "encryptionkey@0"
I20260926 03:05:56.581846 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6fee0d8a1dd14538bbf905a157dfad21"
format_stamp: "Formatted at 2026-09-26 03:05:56 on dist-test-slave-hmbw"
server_key: "8f3482106d9074dec8a78b653f574079"
server_key_iv: "0a94918f294832208ca287d219f2a971"
server_key_version: "encryptionkey@0"
I20260926 03:05:56.587751 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 03:05:56.591585 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:05:56.593143 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:05:56.593338 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "6fee0d8a1dd14538bbf905a157dfad21"
format_stamp: "Formatted at 2026-09-26 03:05:56 on dist-test-slave-hmbw"
server_key: "8f3482106d9074dec8a78b653f574079"
server_key_iv: "0a94918f294832208ca287d219f2a971"
server_key_version: "encryptionkey@0"
I20260926 03:05:56.593544 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:05:56.609249 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:05:56.614032 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:05:56.614378 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:05:56.615746 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:05:56.617331 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:05:56.617424 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:05:56.617544 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:05:56.617599 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:05:56.649713 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42079
I20260926 03:05:56.649760 1151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42079 every 8 connection(s)
I20260926 03:05:56.651068 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:05:56.652908 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 1024
I20260926 03:05:56.653116 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:05:56.653741 388 external_mini_cluster.cc:1428] Setting key a51ea83a47ba5ef4e28da14f157d6a53
Sep 26 03:05:56 dist-test-slave-hmbw krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 03:05:56.673925 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:56.653775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37015 (local address 127.0.97.62:39191)
0926 03:05:56.654258 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:56.654265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:56.655626 (+ 1361us) server_negotiation.cc:407] Connection header received
0926 03:05:56.657288 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:56.657294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:56.657375 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:56.657478 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:56.659120 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:56.660209 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:56.662097 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:56.662709 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:56.662843 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:56.667210 (+ 4367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:56.667220 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:56.667225 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:56.667274 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:56.670364 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:56.671371 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:56.671376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:56.671378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:56.671592 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:56.672107 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:56.672110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:56.672112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:56.672382 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:56.672506 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:56.673477 (+ 971us) server_negotiation.cc:299] Negotiation successful
0926 03:05:56.673664 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":223,"threads_started":1}
I20260926 03:05:56.675300 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:56.654381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39191 (local address 127.0.97.1:37015)
0926 03:05:56.655307 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:56.655356 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:56.657008 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:56.657733 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:56.657742 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:56.658269 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:05:56.658934 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:56.658961 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:56.660439 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:56.660444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:56.661917 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:56.661933 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:56.662996 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:56.663003 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:56.663237 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:56.664065 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:05:56.664091 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:05:56.666895 (+ 2804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:05:56.670588 (+ 3693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:56.670597 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:56.670612 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:56.671201 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:56.671725 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:56.671729 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:56.671731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:56.671993 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:56.672512 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:05:56.672521 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:05:56.673009 (+ 488us) client_negotiation.cc:769] Sending connection context
0926 03:05:56.673306 (+ 297us) client_negotiation.cc:240] Negotiation successful
0926 03:05:56.673875 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":349,"threads_started":1}
I20260926 03:05:56.677007 1152 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39191
I20260926 03:05:56.677399 1152 heartbeater.cc:464] Registering TS with master...
I20260926 03:05:56.678133 1152 heartbeater.cc:511] Master 127.0.97.62:39191 requested a full tablet report, sending...
I20260926 03:05:56.680335 963 ts_manager.cc:194] Registered new tserver with Master: 6fee0d8a1dd14538bbf905a157dfad21 (127.0.97.1:42079)
I20260926 03:05:56.681797 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:05:56.694326 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39191
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:05:57.020159 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:05:57.020566 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:05:57.020711 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:05:57.030610 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:05:57.030761 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:05:57.030934 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:05:57.042591 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39191
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:05:57.044178 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:05:57.045586 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:05:57.060040 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:57.060077 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:57.060118 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:05:57.063134 1158 server_base.cc:1034] running on GCE node
I20260926 03:05:57.063985 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:05:57.065620 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:05:57.066921 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790391957066863 us; error 53 us; skew 500 ppm
I20260926 03:05:57.067619 1158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:05:57 dist-test-slave-hmbw krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:05:57.071267 1158 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:05:57.073416 1158 webserver.cc:460] Webserver started at http://127.0.97.2:45677/ using document root <none> and password file <none>
I20260926 03:05:57.074388 1158 fs_manager.cc:359] Metadata directory not provided
I20260926 03:05:57.074760 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:05:57.075156 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:05:57.078608 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-1/data/instance:
uuid: "1354016d7fdc44b19a70d1b3889a219f"
format_stamp: "Formatted at 2026-09-26 03:05:57 on dist-test-slave-hmbw"
server_key: "48497f17e6a5831221a74c41ccc733c8"
server_key_iv: "6b2f2bb025708f3d5e7d1d78f9879c4e"
server_key_version: "encryptionkey@0"
I20260926 03:05:57.079499 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1354016d7fdc44b19a70d1b3889a219f"
format_stamp: "Formatted at 2026-09-26 03:05:57 on dist-test-slave-hmbw"
server_key: "48497f17e6a5831221a74c41ccc733c8"
server_key_iv: "6b2f2bb025708f3d5e7d1d78f9879c4e"
server_key_version: "encryptionkey@0"
I20260926 03:05:57.085776 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 03:05:57.089921 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:05:57.091841 1158 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:05:57.092046 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "1354016d7fdc44b19a70d1b3889a219f"
format_stamp: "Formatted at 2026-09-26 03:05:57 on dist-test-slave-hmbw"
server_key: "48497f17e6a5831221a74c41ccc733c8"
server_key_iv: "6b2f2bb025708f3d5e7d1d78f9879c4e"
server_key_version: "encryptionkey@0"
I20260926 03:05:57.092283 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:05:57.133970 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:05:57.138756 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:05:57.139012 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:05:57.140055 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:05:57.141456 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:05:57.141537 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:05:57.141620 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:05:57.141660 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:05:57.175330 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32925
I20260926 03:05:57.175393 1285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32925 every 8 connection(s)
I20260926 03:05:57.176668 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:05:57.182241 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 1158
I20260926 03:05:57.182427 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:05:57.182861 388 external_mini_cluster.cc:1428] Setting key 6263553dcc8fa9380b8d666be6ed19e2
Sep 26 03:05:57 dist-test-slave-hmbw krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 03:05:57.198280 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:57.179358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36535 (local address 127.0.97.62:39191)
0926 03:05:57.179790 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:57.179794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:57.180852 (+ 1058us) server_negotiation.cc:407] Connection header received
0926 03:05:57.182761 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:57.182767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:57.182853 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:57.182952 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:57.184702 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.185434 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.187121 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.187583 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.187701 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:57.191715 (+ 4014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:57.191721 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:57.191725 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:57.191761 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:57.194323 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:57.195471 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:57.195477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:57.195482 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:57.195652 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:57.196228 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:57.196233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:57.196237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:57.196620 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:57.196771 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:57.197638 (+ 867us) server_negotiation.cc:299] Negotiation successful
0926 03:05:57.197893 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":214,"threads_started":1}
I20260926 03:05:57.199524 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:57.179940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39191 (local address 127.0.97.2:36535)
0926 03:05:57.180681 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:57.180726 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:57.182420 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:57.183218 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:57.183227 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:57.183741 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:05:57.184534 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:57.184568 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.185597 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.185603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:57.186972 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:57.186985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.187820 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.187827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:57.188067 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:57.188778 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:05:57.188802 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:05:57.191405 (+ 2603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:05:57.194715 (+ 3310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:57.194724 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:57.194740 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:57.195226 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:57.195830 (+ 604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:57.195836 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:57.195839 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:57.196084 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:57.196767 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:05:57.196774 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:05:57.197182 (+ 408us) client_negotiation.cc:769] Sending connection context
0926 03:05:57.197505 (+ 323us) client_negotiation.cc:240] Negotiation successful
0926 03:05:57.198199 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":292,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 03:05:57.201501 1286 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39191
I20260926 03:05:57.201830 1286 heartbeater.cc:464] Registering TS with master...
I20260926 03:05:57.202656 1286 heartbeater.cc:511] Master 127.0.97.62:39191 requested a full tablet report, sending...
I20260926 03:05:57.204262 963 ts_manager.cc:194] Registered new tserver with Master: 1354016d7fdc44b19a70d1b3889a219f (127.0.97.2:32925)
I20260926 03:05:57.205174 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:05:57.226518 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39191
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:05:57.558132 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:05:57.558426 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:05:57.558568 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:05:57.566011 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:05:57.566145 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:05:57.566299 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:05:57.576340 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39191
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:05:57.577665 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:05:57.579053 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:05:57.592763 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:57.592784 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:57.592761 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:05:57.593961 1292 server_base.cc:1034] running on GCE node
I20260926 03:05:57.594729 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:05:57.595795 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:05:57.597007 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790391957596952 us; error 52 us; skew 500 ppm
I20260926 03:05:57.597353 1292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:05:57 dist-test-slave-hmbw krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:05:57.600816 1292 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:05:57.602396 1292 webserver.cc:460] Webserver started at http://127.0.97.3:39803/ using document root <none> and password file <none>
I20260926 03:05:57.603124 1292 fs_manager.cc:359] Metadata directory not provided
I20260926 03:05:57.603259 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:05:57.603574 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:05:57.606657 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-2/data/instance:
uuid: "d7c4062109b749a79fecc54ce2d3929f"
format_stamp: "Formatted at 2026-09-26 03:05:57 on dist-test-slave-hmbw"
server_key: "5d5d66bcc2e7e5d75c41436313026850"
server_key_iv: "dbcd2a2915b4441c4df2eb0a7fa23f2f"
server_key_version: "encryptionkey@0"
I20260926 03:05:57.607467 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d7c4062109b749a79fecc54ce2d3929f"
format_stamp: "Formatted at 2026-09-26 03:05:57 on dist-test-slave-hmbw"
server_key: "5d5d66bcc2e7e5d75c41436313026850"
server_key_iv: "dbcd2a2915b4441c4df2eb0a7fa23f2f"
server_key_version: "encryptionkey@0"
I20260926 03:05:57.613003 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:05:57.617051 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:05:57.618654 1292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:05:57.618846 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "d7c4062109b749a79fecc54ce2d3929f"
format_stamp: "Formatted at 2026-09-26 03:05:57 on dist-test-slave-hmbw"
server_key: "5d5d66bcc2e7e5d75c41436313026850"
server_key_iv: "dbcd2a2915b4441c4df2eb0a7fa23f2f"
server_key_version: "encryptionkey@0"
I20260926 03:05:57.619047 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:05:57.646368 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:05:57.650838 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:05:57.651106 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:05:57.652086 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:05:57.653535 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:05:57.653622 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:05:57.653748 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:05:57.653803 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:05:57.684973 1152 heartbeater.cc:503] Master 127.0.97.62:39191 was elected leader, sending a full tablet report...
I20260926 03:05:57.687264 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34711
I20260926 03:05:57.687305 1419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34711 every 8 connection(s)
I20260926 03:05:57.688539 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:05:57.694398 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 1292
I20260926 03:05:57.694643 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391950058021-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:05:57.695145 388 external_mini_cluster.cc:1428] Setting key 77774c96e8cdcffd766b69493928427a
Sep 26 03:05:57 dist-test-slave-hmbw krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:05:57.712586 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:57.691617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56067 (local address 127.0.97.62:39191)
0926 03:05:57.691866 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:57.691872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:57.693221 (+ 1349us) server_negotiation.cc:407] Connection header received
0926 03:05:57.695772 (+ 2551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:57.695779 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:57.695895 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:57.696039 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:57.698293 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.699355 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.701145 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.701574 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.701685 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:57.705686 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:57.705693 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:57.705696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:57.705736 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:57.708479 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:57.709499 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:57.709505 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:57.709507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:57.709656 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:57.710181 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:57.710186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:57.710188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:57.710877 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:57.711043 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:57.712018 (+ 975us) server_negotiation.cc:299] Negotiation successful
0926 03:05:57.712251 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260926 03:05:57.714192 1422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:57.692162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39191 (local address 127.0.97.3:56067)
0926 03:05:57.693023 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:57.693067 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:57.695347 (+ 2280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:57.696475 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:57.696490 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:57.697333 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:05:57.698100 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:57.698128 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.699563 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.699567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:57.700951 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:57.700965 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.701866 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.701873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:57.702113 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:57.703039 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:05:57.703073 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:05:57.705381 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:05:57.708682 (+ 3301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:57.708690 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:57.708707 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:57.709259 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:57.709788 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:57.709793 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:57.709811 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:57.710050 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:57.711049 (+ 999us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:05:57.711057 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:05:57.711541 (+ 484us) client_negotiation.cc:769] Sending connection context
0926 03:05:57.711870 (+ 329us) client_negotiation.cc:240] Negotiation successful
0926 03:05:57.712434 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":371,"threads_started":1}
I20260926 03:05:57.715857 1420 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39191
I20260926 03:05:57.716171 1420 heartbeater.cc:464] Registering TS with master...
I20260926 03:05:57.716966 1420 heartbeater.cc:511] Master 127.0.97.62:39191 requested a full tablet report, sending...
I20260926 03:05:57.718830 963 ts_manager.cc:194] Registered new tserver with Master: d7c4062109b749a79fecc54ce2d3929f (127.0.97.3:34711)
I20260926 03:05:57.719700 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56067
I20260926 03:05:57.724901 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:05:57.745688 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:57.731308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60244 (local address 127.0.97.62:39191)
0926 03:05:57.731491 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:57.731498 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:57.732036 (+ 538us) server_negotiation.cc:407] Connection header received
0926 03:05:57.732202 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:57.732206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:57.732273 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:57.732368 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:57.733971 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.734824 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.736177 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.736619 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.736740 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:57.739833 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:57.739840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:57.739843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:57.739886 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:57.742724 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:57.743512 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:57.743519 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:57.743521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:57.743653 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:57.744034 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:57.744039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:57.744041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:57.744428 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:57.744659 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:57.745063 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 03:05:57.745349 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:05:57.753134 963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:05:57.760630 963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:05:57.800673 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:57.777067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54068 (local address 127.0.97.1:42079)
0926 03:05:57.790279 (+ 13212us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:57.790288 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:57.790318 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:05:57.790435 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:57.790445 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:57.790779 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:57.791768 (+ 989us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:05:57.792028 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.793635 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.797699 (+ 4064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.799370 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.799533 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:57.799787 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:57.800150 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 03:05:57.800335 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13036,"thread_start_us":12878,"threads_started":1}
I20260926 03:05:57.800832 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:57.776835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42079 (local address 127.0.0.1:54068)
0926 03:05:57.787716 (+ 10881us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:57.787769 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:57.788037 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:57.791057 (+ 3020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:57.791069 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:57.791101 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:05:57.791558 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:57.791568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.793794 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.793799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:57.797177 (+ 3378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:57.797201 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.799615 (+ 2414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.799623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:57.799858 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:57.799916 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 03:05:57.800097 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 03:05:57.800320 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10530,"mutex_wait_us":215,"spinlock_wait_cycles":3968,"thread_start_us":13946,"threads_started":1}
I20260926 03:05:57.801235 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:57.777285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34711 (local address 127.0.0.1:35550)
0926 03:05:57.791077 (+ 13792us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:57.791095 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:57.791213 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:57.791804 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:57.791808 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:57.791835 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:05:57.792711 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:57.792726 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.795153 (+ 2427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.795160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:57.798033 (+ 2873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:57.798048 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.800314 (+ 2266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.800321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:57.800591 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:57.800632 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 03:05:57.800734 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 03:05:57.800903 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13671,"thread_start_us":11041,"threads_started":1}
I20260926 03:05:57.801977 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:57.777425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54912 (local address 127.0.97.2:32925)
0926 03:05:57.790617 (+ 13192us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:57.790627 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:57.790661 (+ 34us) server_negotiation.cc:407] Connection header received
0926 03:05:57.790775 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:57.790784 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:57.791022 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:57.791994 (+ 972us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:05:57.792173 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.793494 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.797735 (+ 4241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.799380 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.799533 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:57.799754 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:57.801119 (+ 1365us) server_negotiation.cc:299] Negotiation successful
0926 03:05:57.801302 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13010,"thread_start_us":12915,"threads_started":1}
I20260926 03:05:57.802105 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:57.776969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32925 (local address 127.0.0.1:54912)
0926 03:05:57.789543 (+ 12574us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:57.789563 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:57.789687 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:57.791252 (+ 1565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:57.791261 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:57.791294 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:05:57.791786 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:57.791800 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.793665 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.793670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:57.797177 (+ 3507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:57.797205 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.800641 (+ 3436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.800647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:57.800830 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:57.800861 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 03:05:57.800948 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 03:05:57.801093 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12443,"mutex_wait_us":187,"thread_start_us":12279,"threads_started":1}
I20260926 03:05:57.803272 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:57.777873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35550 (local address 127.0.97.3:34711)
0926 03:05:57.787895 (+ 10022us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:57.787905 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:57.791215 (+ 3310us) server_negotiation.cc:407] Connection header received
0926 03:05:57.791349 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:57.791357 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:57.791589 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:57.791792 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:05:57.792935 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.794837 (+ 1902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.798439 (+ 3602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.800101 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.800241 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:57.800469 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:57.802440 (+ 1971us) server_negotiation.cc:299] Negotiation successful
0926 03:05:57.802905 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9849,"thread_start_us":9670,"threads_started":1}
I20260926 03:05:57.806380 1221 tablet_service.cc:1467] Processing CreateTablet for tablet a7e86d10357e4315aa7d018d5ea693af (DEFAULT_TABLE table=test-table [id=5ad8e0b6380a4bda960390fafcc7e0be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:05:57.806371 1087 tablet_service.cc:1467] Processing CreateTablet for tablet a7e86d10357e4315aa7d018d5ea693af (DEFAULT_TABLE table=test-table [id=5ad8e0b6380a4bda960390fafcc7e0be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:05:57.806356 1355 tablet_service.cc:1467] Processing CreateTablet for tablet a7e86d10357e4315aa7d018d5ea693af (DEFAULT_TABLE table=test-table [id=5ad8e0b6380a4bda960390fafcc7e0be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:05:57.808110 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7e86d10357e4315aa7d018d5ea693af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:05:57.808110 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7e86d10357e4315aa7d018d5ea693af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:05:57.808110 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7e86d10357e4315aa7d018d5ea693af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:05:57.824638 1438 tablet_bootstrap.cc:492] T a7e86d10357e4315aa7d018d5ea693af P 1354016d7fdc44b19a70d1b3889a219f: Bootstrap starting.
I20260926 03:05:57.832779 1438 tablet_bootstrap.cc:654] T a7e86d10357e4315aa7d018d5ea693af P 1354016d7fdc44b19a70d1b3889a219f: Neither blocks nor log segments found. Creating new log.
I20260926 03:05:57.832769 1439 tablet_bootstrap.cc:492] T a7e86d10357e4315aa7d018d5ea693af P d7c4062109b749a79fecc54ce2d3929f: Bootstrap starting.
I20260926 03:05:57.834980 1438 log.cc:824] T a7e86d10357e4315aa7d018d5ea693af P 1354016d7fdc44b19a70d1b3889a219f: Log is configured to *not* fsync() on all Append() calls
I20260926 03:05:57.835522 1440 tablet_bootstrap.cc:492] T a7e86d10357e4315aa7d018d5ea693af P 6fee0d8a1dd14538bbf905a157dfad21: Bootstrap starting.
I20260926 03:05:57.837426 1439 tablet_bootstrap.cc:654] T a7e86d10357e4315aa7d018d5ea693af P d7c4062109b749a79fecc54ce2d3929f: Neither blocks nor log segments found. Creating new log.
I20260926 03:05:57.839073 1439 log.cc:824] T a7e86d10357e4315aa7d018d5ea693af P d7c4062109b749a79fecc54ce2d3929f: Log is configured to *not* fsync() on all Append() calls
I20260926 03:05:57.839314 1440 tablet_bootstrap.cc:654] T a7e86d10357e4315aa7d018d5ea693af P 6fee0d8a1dd14538bbf905a157dfad21: Neither blocks nor log segments found. Creating new log.
I20260926 03:05:57.840058 1438 tablet_bootstrap.cc:492] T a7e86d10357e4315aa7d018d5ea693af P 1354016d7fdc44b19a70d1b3889a219f: No bootstrap required, opened a new log
I20260926 03:05:57.840355 1438 ts_tablet_manager.cc:1397] T a7e86d10357e4315aa7d018d5ea693af P 1354016d7fdc44b19a70d1b3889a219f: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s
I20260926 03:05:57.840699 1440 log.cc:824] T a7e86d10357e4315aa7d018d5ea693af P 6fee0d8a1dd14538bbf905a157dfad21: Log is configured to *not* fsync() on all Append() calls
I20260926 03:05:57.843279 1439 tablet_bootstrap.cc:492] T a7e86d10357e4315aa7d018d5ea693af P d7c4062109b749a79fecc54ce2d3929f: No bootstrap required, opened a new log
I20260926 03:05:57.843375 1440 tablet_bootstrap.cc:492] T a7e86d10357e4315aa7d018d5ea693af P 6fee0d8a1dd14538bbf905a157dfad21: No bootstrap required, opened a new log
I20260926 03:05:57.843601 1439 ts_tablet_manager.cc:1397] T a7e86d10357e4315aa7d018d5ea693af P d7c4062109b749a79fecc54ce2d3929f: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.009s
I20260926 03:05:57.843626 1440 ts_tablet_manager.cc:1397] T a7e86d10357e4315aa7d018d5ea693af P 6fee0d8a1dd14538bbf905a157dfad21: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260926 03:05:57.851325 1440 raft_consensus.cc:348] T a7e86d10357e4315aa7d018d5ea693af P 6fee0d8a1dd14538bbf905a157dfad21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fee0d8a1dd14538bbf905a157dfad21" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42079 } } peers { permanent_uuid: "1354016d7fdc44b19a70d1b3889a219f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32925 } } peers { permanent_uuid: "d7c4062109b749a79fecc54ce2d3929f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34711 } }
I20260926 03:05:57.851733 1440 raft_consensus.cc:374] T a7e86d10357e4315aa7d018d5ea693af P 6fee0d8a1dd14538bbf905a157dfad21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:05:57.851823 1440 raft_consensus.cc:729] T a7e86d10357e4315aa7d018d5ea693af P 6fee0d8a1dd14538bbf905a157dfad21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fee0d8a1dd14538bbf905a157dfad21, State: Initialized, Role: FOLLOWER
I20260926 03:05:57.852442 1440 consensus_queue.cc:260] T a7e86d10357e4315aa7d018d5ea693af P 6fee0d8a1dd14538bbf905a157dfad21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fee0d8a1dd14538bbf905a157dfad21" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42079 } } peers { permanent_uuid: "1354016d7fdc44b19a70d1b3889a219f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32925 } } peers { permanent_uuid: "d7c4062109b749a79fecc54ce2d3929f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34711 } }
I20260926 03:05:57.852861 1438 raft_consensus.cc:348] T a7e86d10357e4315aa7d018d5ea693af P 1354016d7fdc44b19a70d1b3889a219f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fee0d8a1dd14538bbf905a157dfad21" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42079 } } peers { permanent_uuid: "1354016d7fdc44b19a70d1b3889a219f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32925 } } peers { permanent_uuid: "d7c4062109b749a79fecc54ce2d3929f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34711 } }
I20260926 03:05:57.853359 1438 raft_consensus.cc:374] T a7e86d10357e4315aa7d018d5ea693af P 1354016d7fdc44b19a70d1b3889a219f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:05:57.853471 1438 raft_consensus.cc:729] T a7e86d10357e4315aa7d018d5ea693af P 1354016d7fdc44b19a70d1b3889a219f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1354016d7fdc44b19a70d1b3889a219f, State: Initialized, Role: FOLLOWER
I20260926 03:05:57.854348 1440 ts_tablet_manager.cc:1428] T a7e86d10357e4315aa7d018d5ea693af P 6fee0d8a1dd14538bbf905a157dfad21: Time spent starting tablet: real 0.011s user 0.002s sys 0.008s
I20260926 03:05:57.854226 1438 consensus_queue.cc:260] T a7e86d10357e4315aa7d018d5ea693af P 1354016d7fdc44b19a70d1b3889a219f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fee0d8a1dd14538bbf905a157dfad21" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42079 } } peers { permanent_uuid: "1354016d7fdc44b19a70d1b3889a219f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32925 } } peers { permanent_uuid: "d7c4062109b749a79fecc54ce2d3929f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34711 } }
I20260926 03:05:57.854883 1439 raft_consensus.cc:348] T a7e86d10357e4315aa7d018d5ea693af P d7c4062109b749a79fecc54ce2d3929f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fee0d8a1dd14538bbf905a157dfad21" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42079 } } peers { permanent_uuid: "1354016d7fdc44b19a70d1b3889a219f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32925 } } peers { permanent_uuid: "d7c4062109b749a79fecc54ce2d3929f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34711 } }
I20260926 03:05:57.855441 1439 raft_consensus.cc:374] T a7e86d10357e4315aa7d018d5ea693af P d7c4062109b749a79fecc54ce2d3929f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:05:57.855525 1439 raft_consensus.cc:729] T a7e86d10357e4315aa7d018d5ea693af P d7c4062109b749a79fecc54ce2d3929f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c4062109b749a79fecc54ce2d3929f, State: Initialized, Role: FOLLOWER
I20260926 03:05:57.855906 1286 heartbeater.cc:503] Master 127.0.97.62:39191 was elected leader, sending a full tablet report...
I20260926 03:05:57.856313 1439 consensus_queue.cc:260] T a7e86d10357e4315aa7d018d5ea693af P d7c4062109b749a79fecc54ce2d3929f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fee0d8a1dd14538bbf905a157dfad21" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42079 } } peers { permanent_uuid: "1354016d7fdc44b19a70d1b3889a219f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32925 } } peers { permanent_uuid: "d7c4062109b749a79fecc54ce2d3929f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34711 } }
I20260926 03:05:57.857879 1420 heartbeater.cc:503] Master 127.0.97.62:39191 was elected leader, sending a full tablet report...
I20260926 03:05:57.858734 1439 ts_tablet_manager.cc:1428] T a7e86d10357e4315aa7d018d5ea693af P d7c4062109b749a79fecc54ce2d3929f: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260926 03:05:57.860189 1438 ts_tablet_manager.cc:1428] T a7e86d10357e4315aa7d018d5ea693af P 1354016d7fdc44b19a70d1b3889a219f: Time spent starting tablet: real 0.020s user 0.008s sys 0.008s
W20260926 03:05:57.903692 1153 tablet.cc:2369] T a7e86d10357e4315aa7d018d5ea693af P 6fee0d8a1dd14538bbf905a157dfad21: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:05:57.928854 1287 tablet.cc:2369] T a7e86d10357e4315aa7d018d5ea693af P 1354016d7fdc44b19a70d1b3889a219f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:05:57.931088 1445 raft_consensus.cc:482] T a7e86d10357e4315aa7d018d5ea693af P 1354016d7fdc44b19a70d1b3889a219f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:05:57.931391 1445 raft_consensus.cc:504] T a7e86d10357e4315aa7d018d5ea693af P 1354016d7fdc44b19a70d1b3889a219f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fee0d8a1dd14538bbf905a157dfad21" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42079 } } peers { permanent_uuid: "1354016d7fdc44b19a70d1b3889a219f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32925 } } peers { permanent_uuid: "d7c4062109b749a79fecc54ce2d3929f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34711 } }
I20260926 03:05:57.933151 1445 leader_election.cc:288] T a7e86d10357e4315aa7d018d5ea693af P 1354016d7fdc44b19a70d1b3889a219f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fee0d8a1dd14538bbf905a157dfad21 (127.0.97.1:42079), d7c4062109b749a79fecc54ce2d3929f (127.0.97.3:34711)
W20260926 03:05:57.940685 1421 tablet.cc:2369] T a7e86d10357e4315aa7d018d5ea693af P d7c4062109b749a79fecc54ce2d3929f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:05:57.941464 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:57.933766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33099 (local address 127.0.97.1:42079)
0926 03:05:57.933976 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:57.933984 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:57.934770 (+ 786us) server_negotiation.cc:407] Connection header received
0926 03:05:57.934990 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:57.934995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:57.935059 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:57.935136 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:05:57.935720 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.936475 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.938555 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.940291 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.940453 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:57.940563 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:57.941059 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 03:05:57.941203 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 03:05:57.941465 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:57.933844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60023 (local address 127.0.97.3:34711)
0926 03:05:57.934036 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:57.934043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:57.934712 (+ 669us) server_negotiation.cc:407] Connection header received
0926 03:05:57.934897 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:57.934902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:57.934971 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:57.935045 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:05:57.935609 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.936556 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.939042 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.940326 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.940453 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:57.940563 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:57.941097 (+ 534us) server_negotiation.cc:299] Negotiation successful
0926 03:05:57.941222 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:05:57.941650 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:57.933693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42079 (local address 127.0.97.2:33099)
0926 03:05:57.934666 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:57.934680 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:57.934858 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:57.935163 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:57.935168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:57.935205 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:05:57.935587 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:57.935596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.936618 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.936623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:57.938326 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:57.938343 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.940644 (+ 2301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.940653 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:57.940857 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:57.940896 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 03:05:57.941011 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 03:05:57.941165 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"mutex_wait_us":1,"thread_start_us":482,"threads_started":1}
I20260926 03:05:57.942183 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:57.933647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34711 (local address 127.0.97.2:60023)
0926 03:05:57.934592 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:57.934609 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:57.934768 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:57.935076 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:57.935081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:57.935104 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:05:57.935467 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:57.935479 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.936839 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.936845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:57.938773 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:57.938794 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.940731 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.940736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:57.940919 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:57.940951 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 03:05:57.941646 (+ 695us) client_negotiation.cc:240] Negotiation successful
0926 03:05:57.941791 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":6,"thread_start_us":575,"threads_started":1}
I20260926 03:05:57.943183 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7e86d10357e4315aa7d018d5ea693af" candidate_uuid: "1354016d7fdc44b19a70d1b3889a219f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fee0d8a1dd14538bbf905a157dfad21" is_pre_election: true
I20260926 03:05:57.943264 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7e86d10357e4315aa7d018d5ea693af" candidate_uuid: "1354016d7fdc44b19a70d1b3889a219f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c4062109b749a79fecc54ce2d3929f" is_pre_election: true
I20260926 03:05:57.944103 1375 raft_consensus.cc:2454] T a7e86d10357e4315aa7d018d5ea693af P d7c4062109b749a79fecc54ce2d3929f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1354016d7fdc44b19a70d1b3889a219f in term 0.
I20260926 03:05:57.944149 1107 raft_consensus.cc:2454] T a7e86d10357e4315aa7d018d5ea693af P 6fee0d8a1dd14538bbf905a157dfad21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1354016d7fdc44b19a70d1b3889a219f in term 0.
I20260926 03:05:57.945045 1174 leader_election.cc:302] T a7e86d10357e4315aa7d018d5ea693af P 1354016d7fdc44b19a70d1b3889a219f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1354016d7fdc44b19a70d1b3889a219f, 6fee0d8a1dd14538bbf905a157dfad21; no voters:
I20260926 03:05:57.945555 1445 raft_consensus.cc:2781] T a7e86d10357e4315aa7d018d5ea693af P 1354016d7fdc44b19a70d1b3889a219f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:05:57.945715 1445 raft_consensus.cc:482] T a7e86d10357e4315aa7d018d5ea693af P 1354016d7fdc44b19a70d1b3889a219f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:05:57.945806 1445 raft_consensus.cc:3037] T a7e86d10357e4315aa7d018d5ea693af P 1354016d7fdc44b19a70d1b3889a219f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:05:57.948202 1445 raft_consensus.cc:504] T a7e86d10357e4315aa7d018d5ea693af P 1354016d7fdc44b19a70d1b3889a219f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fee0d8a1dd14538bbf905a157dfad21" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42079 } } peers { permanent_uuid: "1354016d7fdc44b19a70d1b3889a219f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32925 } } peers { permanent_uuid: "d7c4062109b749a79fecc54ce2d3929f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34711 } }
I20260926 03:05:57.948858 1445 leader_election.cc:288] T a7e86d10357e4315aa7d018d5ea693af P 1354016d7fdc44b19a70d1b3889a219f [CANDIDATE]: Term 1 election: Requested vote from peers 6fee0d8a1dd14538bbf905a157dfad21 (127.0.97.1:42079), d7c4062109b749a79fecc54ce2d3929f (127.0.97.3:34711)
I20260926 03:05:57.949463 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7e86d10357e4315aa7d018d5ea693af" candidate_uuid: "1354016d7fdc44b19a70d1b3889a219f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c4062109b749a79fecc54ce2d3929f"
I20260926 03:05:57.949468 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7e86d10357e4315aa7d018d5ea693af" candidate_uuid: "1354016d7fdc44b19a70d1b3889a219f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fee0d8a1dd14538bbf905a157dfad21"
I20260926 03:05:57.949692 1375 raft_consensus.cc:3037] T a7e86d10357e4315aa7d018d5ea693af P d7c4062109b749a79fecc54ce2d3929f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:05:57.949683 1107 raft_consensus.cc:3037] T a7e86d10357e4315aa7d018d5ea693af P 6fee0d8a1dd14538bbf905a157dfad21 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:05:57.952360 1375 raft_consensus.cc:2454] T a7e86d10357e4315aa7d018d5ea693af P d7c4062109b749a79fecc54ce2d3929f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1354016d7fdc44b19a70d1b3889a219f in term 1.
I20260926 03:05:57.952605 1107 raft_consensus.cc:2454] T a7e86d10357e4315aa7d018d5ea693af P 6fee0d8a1dd14538bbf905a157dfad21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1354016d7fdc44b19a70d1b3889a219f in term 1.
I20260926 03:05:57.953028 1176 leader_election.cc:302] T a7e86d10357e4315aa7d018d5ea693af P 1354016d7fdc44b19a70d1b3889a219f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1354016d7fdc44b19a70d1b3889a219f, d7c4062109b749a79fecc54ce2d3929f; no voters:
I20260926 03:05:57.953395 1445 raft_consensus.cc:2781] T a7e86d10357e4315aa7d018d5ea693af P 1354016d7fdc44b19a70d1b3889a219f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:05:57.953861 1445 raft_consensus.cc:686] T a7e86d10357e4315aa7d018d5ea693af P 1354016d7fdc44b19a70d1b3889a219f [term 1 LEADER]: Becoming Leader. State: Replica: 1354016d7fdc44b19a70d1b3889a219f, State: Running, Role: LEADER
I20260926 03:05:57.954448 1445 consensus_queue.cc:237] T a7e86d10357e4315aa7d018d5ea693af P 1354016d7fdc44b19a70d1b3889a219f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fee0d8a1dd14538bbf905a157dfad21" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42079 } } peers { permanent_uuid: "1354016d7fdc44b19a70d1b3889a219f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32925 } } peers { permanent_uuid: "d7c4062109b749a79fecc54ce2d3929f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34711 } }
I20260926 03:05:57.959884 961 catalog_manager.cc:5495] T a7e86d10357e4315aa7d018d5ea693af P 1354016d7fdc44b19a70d1b3889a219f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1354016d7fdc44b19a70d1b3889a219f (127.0.97.2). New cstate: current_term: 1 leader_uuid: "1354016d7fdc44b19a70d1b3889a219f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fee0d8a1dd14538bbf905a157dfad21" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1354016d7fdc44b19a70d1b3889a219f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7c4062109b749a79fecc54ce2d3929f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34711 } health_report { overall_health: UNKNOWN } } }
Sep 26 03:05:57 dist-test-slave-hmbw krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:05:58.015166 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:57.993792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54072 (local address 127.0.97.1:42079)
0926 03:05:57.993949 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:57.993954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:57.994572 (+ 618us) server_negotiation.cc:407] Connection header received
0926 03:05:57.994739 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:57.994745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:57.994852 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:57.994956 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:57.996627 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:57.998302 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:57.999828 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:58.000517 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:58.000656 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:58.005501 (+ 4845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:58.005523 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:58.005544 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:58.005605 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:58.009588 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:58.011582 (+ 1994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:58.011594 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:58.011597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:58.011997 (+ 400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:58.012620 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:58.012625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:58.012627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:58.013095 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:58.013357 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:58.014292 (+ 935us) server_negotiation.cc:299] Negotiation successful
0926 03:05:58.014817 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 03:05:58 dist-test-slave-hmbw krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391958, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 03:05:58 dist-test-slave-hmbw krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391958, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:05:58.063261 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:58.047845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54086 (local address 127.0.97.1:42079)
0926 03:05:58.048067 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:58.048074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:58.048422 (+ 348us) server_negotiation.cc:407] Connection header received
0926 03:05:58.048608 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:58.048612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:58.048682 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:58.048774 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:58.050195 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:58.051205 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:58.052724 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:58.053212 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:58.053330 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:58.057599 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:58.057605 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:58.057608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:58.057652 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:58.060492 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:58.061273 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:58.061277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:58.061280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:58.061439 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:58.061856 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:58.061860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:58.061862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:58.062150 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:58.062414 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:58.062783 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 03:05:58.063000 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 03:05:58 dist-test-slave-hmbw krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 03:05:58 dist-test-slave-hmbw krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:05:58.109886 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:58.094234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54100 (local address 127.0.97.1:42079)
0926 03:05:58.094541 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:58.094549 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:58.094834 (+ 285us) server_negotiation.cc:407] Connection header received
0926 03:05:58.095061 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:58.095066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:58.095133 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:58.095226 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:58.096928 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:58.098206 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:58.099624 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:58.100113 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:58.100226 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:58.104341 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:58.104347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:58.104351 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:58.104397 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:58.106934 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:58.107790 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:58.107794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:58.107797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:58.107962 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:58.108458 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:58.108469 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:58.108471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:58.108775 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:58.108973 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:58.109321 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 03:05:58.109559 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 03:05:58.111740 1067 tablet_service.cc:2908] Rejecting scan request for tablet a7e86d10357e4315aa7d018d5ea693af: Uninitialized: clean time has not yet been initialized
I20260926 03:05:58.115124 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 1024
I20260926 03:05:58.131361 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:58.116253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54112 (local address 127.0.97.1:42079)
0926 03:05:58.116482 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:58.116487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:58.119034 (+ 2547us) server_negotiation.cc:407] Connection header received
0926 03:05:58.119160 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:58.119164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:58.119249 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:58.119335 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:58.120614 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:58.121344 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:58.122649 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:58.123105 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:58.123219 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:58.125592 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:58.125597 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:58.125600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:58.125640 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:58.128361 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:58.129123 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:58.129128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:58.129130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:58.129336 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:58.129722 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:58.129726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:58.129728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:58.130044 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:58.130261 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:58.130819 (+ 558us) server_negotiation.cc:299] Negotiation successful
0926 03:05:58.131031 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260926 03:05:58.131965 1147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:05:58.220983 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 1024
I20260926 03:05:58.235836 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 1158
Sep 26 03:05:58 dist-test-slave-hmbw krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:05:58.251354 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:58.236632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54928 (local address 127.0.97.2:32925)
0926 03:05:58.236885 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:58.236891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:58.236911 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:05:58.237079 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:58.237085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:58.237159 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:58.237241 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:58.238444 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:58.239212 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:58.240490 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:58.240902 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:58.241011 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:58.245289 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:58.245307 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:58.245323 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:58.245363 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:58.248243 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:58.249085 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:58.249093 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:58.249095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:58.249251 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:58.249658 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:58.249662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:58.249664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:58.250051 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:58.250271 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:58.250758 (+ 487us) server_negotiation.cc:299] Negotiation successful
0926 03:05:58.251070 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:05:58.252139 1281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:05:58.341936 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 1158
I20260926 03:05:58.355304 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 1292
Sep 26 03:05:58 dist-test-slave-hmbw krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:05:58.370823 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:58.356040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35562 (local address 127.0.97.3:34711)
0926 03:05:58.356231 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:58.356236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:58.356256 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:05:58.356436 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:58.356441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:58.356517 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:58.356602 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:58.357830 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:58.358641 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:58.359933 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:58.360309 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:58.360425 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:58.364675 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:58.364691 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:58.364708 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:58.364750 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:58.368115 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:58.368896 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:58.368904 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:58.368906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:58.369067 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:58.369422 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:58.369426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:58.369428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:58.369801 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:58.370031 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:58.370326 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 03:05:58.370549 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:05:58.371534 1415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:05:58.458067 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 1292
I20260926 03:05:58.471088 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 931
I20260926 03:05:58.472301 993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:05:58.579763 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 931
2026-09-26T03:05:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4546 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:05:58 dist-test-slave-hmbw krb5kdc[1468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:05:58 dist-test-slave-hmbw krb5kdc[1468](info): set up 2 sockets
Sep 26 03:05:58 dist-test-slave-hmbw krb5kdc[1468](info): commencing operation
krb5kdc: starting...
W20260926 03:05:59.590389 388 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260926 03:05:59.791141 388 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.118s user 0.000s sys 0.004s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/test-user.keytab.
W20260926 03:06:00.244481 388 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.453s user 0.000s sys 0.004s
Sep 26 03:06:00 dist-test-slave-hmbw krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:06:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:06:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:06:00.327955 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39621
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:00.658998 1481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:00.659314 1481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:00.659426 1481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:00.667395 1481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:06:00.667526 1481 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 03:06:00.667567 1481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:00.667646 1481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:00.667713 1481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:06:00.667798 1481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:06:00.678086 1481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39621
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:00.679400 1481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:00.680783 1481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:00.692580 1486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:00.692641 1489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:00.692659 1487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:00.693300 1481 server_base.cc:1034] running on GCE node
I20260926 03:06:00.694051 1481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:00.695662 1481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:00.696893 1481 hybrid_clock.cc:648] HybridClock initialized: now 1790391960696821 us; error 45 us; skew 500 ppm
I20260926 03:06:00.697217 1481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:00 dist-test-slave-hmbw krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:00.705703 1481 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:06:00.707453 1481 webserver.cc:460] Webserver started at http://127.0.97.62:43865/ using document root <none> and password file <none>
I20260926 03:06:00.708230 1481 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:00.708326 1481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:00.708665 1481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:00.711809 1481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "745f1df3f2994915bcd8fe7b399f03ae"
format_stamp: "Formatted at 2026-09-26 03:06:00 on dist-test-slave-hmbw"
server_key: "0ab355b1ea5d59a08ad268f0b9b02846"
server_key_iv: "37f141d26a8155444b8a5016d330ee6f"
server_key_version: "encryptionkey@0"
I20260926 03:06:00.712643 1481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "745f1df3f2994915bcd8fe7b399f03ae"
format_stamp: "Formatted at 2026-09-26 03:06:00 on dist-test-slave-hmbw"
server_key: "0ab355b1ea5d59a08ad268f0b9b02846"
server_key_iv: "37f141d26a8155444b8a5016d330ee6f"
server_key_version: "encryptionkey@0"
I20260926 03:06:00.718303 1481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:06:00.722309 1496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:00.723887 1481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:06:00.724056 1481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "745f1df3f2994915bcd8fe7b399f03ae"
format_stamp: "Formatted at 2026-09-26 03:06:00 on dist-test-slave-hmbw"
server_key: "0ab355b1ea5d59a08ad268f0b9b02846"
server_key_iv: "37f141d26a8155444b8a5016d330ee6f"
server_key_version: "encryptionkey@0"
I20260926 03:06:00.724274 1481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:00.754163 1481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:00.758761 1481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:00.759032 1481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:00.780387 1481 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39621
I20260926 03:06:00.780395 1557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39621 every 8 connection(s)
I20260926 03:06:00.781760 1481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:06:00.785606 1558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:00.786464 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 1481
I20260926 03:06:00.786708 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:06:00.787359 388 external_mini_cluster.cc:1428] Setting key 20997f9bc077738aa0f842da939a026c
I20260926 03:06:00.800900 1558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 745f1df3f2994915bcd8fe7b399f03ae: Bootstrap starting.
Sep 26 03:06:00 dist-test-slave-hmbw krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:00.805104 1558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 745f1df3f2994915bcd8fe7b399f03ae: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:00.806322 1558 log.cc:824] T 00000000000000000000000000000000 P 745f1df3f2994915bcd8fe7b399f03ae: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:00.809948 1558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 745f1df3f2994915bcd8fe7b399f03ae: No bootstrap required, opened a new log
I20260926 03:06:00.814107 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:00.789602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44354 (local address 127.0.97.62:39621)
0926 03:06:00.790469 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:00.790482 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:00.790577 (+ 95us) server_negotiation.cc:407] Connection header received
0926 03:06:00.792245 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:00.792274 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:00.792749 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:00.793113 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:00.795026 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:00.796611 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:00.798553 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:00.799436 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:00.799631 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:00.804794 (+ 5163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:00.804804 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:00.804822 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:00.804877 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:00.808502 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:00.809477 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:00.809485 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:00.809492 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:00.809705 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:00.810148 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:00.810165 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:00.810167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:00.810929 (+ 762us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:00.811317 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:00.812035 (+ 718us) server_negotiation.cc:299] Negotiation successful
0926 03:06:00.812545 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":637,"thread_start_us":397,"threads_started":1}
I20260926 03:06:00.819657 1558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 745f1df3f2994915bcd8fe7b399f03ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745f1df3f2994915bcd8fe7b399f03ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39621 } }
I20260926 03:06:00.820154 1558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 745f1df3f2994915bcd8fe7b399f03ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:00.820282 1558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 745f1df3f2994915bcd8fe7b399f03ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745f1df3f2994915bcd8fe7b399f03ae, State: Initialized, Role: FOLLOWER
I20260926 03:06:00.820958 1558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 745f1df3f2994915bcd8fe7b399f03ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745f1df3f2994915bcd8fe7b399f03ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39621 } }
I20260926 03:06:00.821249 1558 raft_consensus.cc:388] T 00000000000000000000000000000000 P 745f1df3f2994915bcd8fe7b399f03ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:06:00.821378 1558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 745f1df3f2994915bcd8fe7b399f03ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:06:00.821537 1558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 745f1df3f2994915bcd8fe7b399f03ae [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:00.823782 1558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 745f1df3f2994915bcd8fe7b399f03ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745f1df3f2994915bcd8fe7b399f03ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39621 } }
I20260926 03:06:00.824447 1558 leader_election.cc:302] T 00000000000000000000000000000000 P 745f1df3f2994915bcd8fe7b399f03ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 745f1df3f2994915bcd8fe7b399f03ae; no voters:
I20260926 03:06:00.825104 1558 leader_election.cc:288] T 00000000000000000000000000000000 P 745f1df3f2994915bcd8fe7b399f03ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:06:00.825238 1563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 745f1df3f2994915bcd8fe7b399f03ae [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:00.825946 1563 raft_consensus.cc:686] T 00000000000000000000000000000000 P 745f1df3f2994915bcd8fe7b399f03ae [term 1 LEADER]: Becoming Leader. State: Replica: 745f1df3f2994915bcd8fe7b399f03ae, State: Running, Role: LEADER
I20260926 03:06:00.826567 1563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 745f1df3f2994915bcd8fe7b399f03ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745f1df3f2994915bcd8fe7b399f03ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39621 } }
I20260926 03:06:00.827224 1558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 745f1df3f2994915bcd8fe7b399f03ae [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:06:00.829582 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 745f1df3f2994915bcd8fe7b399f03ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "745f1df3f2994915bcd8fe7b399f03ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745f1df3f2994915bcd8fe7b399f03ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39621 } } }
I20260926 03:06:00.830034 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 745f1df3f2994915bcd8fe7b399f03ae [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:00.830991 1569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:06:00.830826 1565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 745f1df3f2994915bcd8fe7b399f03ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 745f1df3f2994915bcd8fe7b399f03ae. Latest consensus state: current_term: 1 leader_uuid: "745f1df3f2994915bcd8fe7b399f03ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745f1df3f2994915bcd8fe7b399f03ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39621 } } }
I20260926 03:06:00.831387 1565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 745f1df3f2994915bcd8fe7b399f03ae [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:00.837347 1569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:06:00.848748 1569 catalog_manager.cc:1347] Generated new cluster ID: d07b146fa5e045beb8014d0b85d2e228
I20260926 03:06:00.848886 1569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:06:00.859750 1569 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:06:00.861166 1569 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:06:00.870007 1569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 745f1df3f2994915bcd8fe7b399f03ae: Generated new TSK 0
I20260926 03:06:00.870942 1569 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:06:00.922919 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39621
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:01.290702 1584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:01.291054 1584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:01.291209 1584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:01.298906 1584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:01.299050 1584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:01.299170 1584 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:06:01.299280 1584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:06:01.310714 1584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:01.312078 1584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:01.313480 1584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:01.327746 1589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:01.327853 1592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:01.328084 1590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:01.329340 1584 server_base.cc:1034] running on GCE node
I20260926 03:06:01.330106 1584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:01.331167 1584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:01.332420 1584 hybrid_clock.cc:648] HybridClock initialized: now 1790391961332358 us; error 46 us; skew 500 ppm
I20260926 03:06:01.332754 1584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:01 dist-test-slave-hmbw krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:01.336426 1584 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:06:01.338080 1584 webserver.cc:460] Webserver started at http://127.0.97.1:36917/ using document root <none> and password file <none>
I20260926 03:06:01.338959 1584 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:01.339075 1584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:01.339461 1584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:01.342466 1584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-0/data/instance:
uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9"
format_stamp: "Formatted at 2026-09-26 03:06:01 on dist-test-slave-hmbw"
server_key: "406d187f14e0740d730cc9ef8c2a0753"
server_key_iv: "77bd9d0a37afadfba3cf9b46afd2ba9f"
server_key_version: "encryptionkey@0"
I20260926 03:06:01.343370 1584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9"
format_stamp: "Formatted at 2026-09-26 03:06:01 on dist-test-slave-hmbw"
server_key: "406d187f14e0740d730cc9ef8c2a0753"
server_key_iv: "77bd9d0a37afadfba3cf9b46afd2ba9f"
server_key_version: "encryptionkey@0"
I20260926 03:06:01.348912 1584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:06:01.352769 1599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:01.354247 1584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:06:01.354431 1584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9"
format_stamp: "Formatted at 2026-09-26 03:06:01 on dist-test-slave-hmbw"
server_key: "406d187f14e0740d730cc9ef8c2a0753"
server_key_iv: "77bd9d0a37afadfba3cf9b46afd2ba9f"
server_key_version: "encryptionkey@0"
I20260926 03:06:01.354710 1584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:01.366281 1584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:01.370865 1584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:01.371116 1584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:01.374135 1584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:01.374243 1584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:01.374370 1584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:01.374425 1584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:06:01 dist-test-slave-hmbw krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:01.398191 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:01.374897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52923 (local address 127.0.97.62:39621)
0926 03:06:01.375313 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:01.375320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:01.376574 (+ 1254us) server_negotiation.cc:407] Connection header received
0926 03:06:01.379177 (+ 2603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:01.379184 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:01.379290 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:01.379388 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:01.381762 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:01.382947 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:01.385230 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:01.385792 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:01.385910 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:01.390930 (+ 5020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:01.390938 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:01.390943 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:01.390989 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:01.394475 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:01.395398 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:01.395405 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:01.395409 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:01.395570 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:01.396161 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:01.396164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:01.396166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:01.396441 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:01.396552 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:01.397737 (+ 1185us) server_negotiation.cc:299] Negotiation successful
0926 03:06:01.397926 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":200,"threads_started":1}
I20260926 03:06:01.400133 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:01.375501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39621 (local address 127.0.97.1:52923)
0926 03:06:01.376377 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:01.376430 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:01.378667 (+ 2237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:01.379746 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:01.379758 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:01.380602 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:01.381521 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:01.381552 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:01.383164 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:01.383170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:01.385077 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:01.385092 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:01.386049 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:01.386058 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:01.386394 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:01.387283 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:01.387315 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:01.390545 (+ 3230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:01.394710 (+ 4165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:01.394722 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:01.394771 (+ 49us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:01.395267 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:01.395713 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:01.395720 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:01.395724 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:01.396046 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:01.396564 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:01.396573 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:01.397242 (+ 669us) client_negotiation.cc:769] Sending connection context
0926 03:06:01.397639 (+ 397us) client_negotiation.cc:240] Negotiation successful
0926 03:06:01.398278 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":371,"threads_started":1}
I20260926 03:06:01.417907 1584 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41143
I20260926 03:06:01.417919 1719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41143 every 8 connection(s)
I20260926 03:06:01.419615 1584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:06:01.421855 1720 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39621
I20260926 03:06:01.422102 1720 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:01.422762 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:01.407279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43177 (local address 127.0.97.62:39621)
0926 03:06:01.407505 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:01.407512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:01.408552 (+ 1040us) server_negotiation.cc:407] Connection header received
0926 03:06:01.408784 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:01.408790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:01.408883 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:01.408984 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:01.410094 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:01.411251 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:01.413221 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:01.413904 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:01.414041 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:01.416344 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:01.416351 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:01.416356 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:01.416400 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:01.419665 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:01.420561 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:01.420568 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:01.420571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:01.420783 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:01.421189 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:01.421194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:01.421197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:01.421590 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:01.421733 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:01.422273 (+ 540us) server_negotiation.cc:299] Negotiation successful
0926 03:06:01.422449 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:06:01.422966 1692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:01.407431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39621 (local address 127.0.97.1:43177)
0926 03:06:01.408406 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:01.408432 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:01.408623 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:01.409046 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:01.409051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:01.409606 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:01.409928 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:01.409940 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:01.411418 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:01.411424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:01.413011 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:01.413030 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:01.414120 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:01.414127 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:01.414320 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:01.415053 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:01.415095 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:01.416119 (+ 1024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:01.419858 (+ 3739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:01.419863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:01.419865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:01.420390 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:01.420923 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:01.420928 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:01.420932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:01.421073 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:01.421768 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:01.421773 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:01.421975 (+ 202us) client_negotiation.cc:769] Sending connection context
0926 03:06:01.422150 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 03:06:01.422490 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"thread_start_us":697,"threads_started":1}
I20260926 03:06:01.423054 1720 heartbeater.cc:511] Master 127.0.97.62:39621 requested a full tablet report, sending...
I20260926 03:06:01.423462 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 1584
I20260926 03:06:01.423699 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:06:01.424440 388 external_mini_cluster.cc:1428] Setting key 6a4732553eca5e275926e3c5a6002d79
I20260926 03:06:01.425210 1513 ts_manager.cc:194] Registered new tserver with Master: bcc8fba247f64f4c8d6d4e2aaac951f9 (127.0.97.1:41143)
I20260926 03:06:01.427115 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52923
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:06:01.466727 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39621
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:01.793578 1724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:01.793879 1724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:01.794005 1724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:01.801606 1724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:01.801730 1724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:01.801838 1724 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:06:01.801935 1724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:06:01.812674 1724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:01.814291 1724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:01.815894 1724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:01.829638 1730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:01.829607 1729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:01.829607 1732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:01.830767 1724 server_base.cc:1034] running on GCE node
I20260926 03:06:01.831408 1724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:01.832366 1724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:01.833721 1724 hybrid_clock.cc:648] HybridClock initialized: now 1790391961833542 us; error 51 us; skew 500 ppm
I20260926 03:06:01.834064 1724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:01 dist-test-slave-hmbw krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:01.837255 1724 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:06:01.838913 1724 webserver.cc:460] Webserver started at http://127.0.97.2:35169/ using document root <none> and password file <none>
I20260926 03:06:01.839635 1724 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:01.839751 1724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:01.840072 1724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:01.843124 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-1/data/instance:
uuid: "b871109b89de4856a53da7e7cc49af2b"
format_stamp: "Formatted at 2026-09-26 03:06:01 on dist-test-slave-hmbw"
server_key: "0c9eea0257f87a866274bb8ea2f6b86b"
server_key_iv: "79c0e3b63c1275384138a57ddc47fca3"
server_key_version: "encryptionkey@0"
I20260926 03:06:01.843976 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b871109b89de4856a53da7e7cc49af2b"
format_stamp: "Formatted at 2026-09-26 03:06:01 on dist-test-slave-hmbw"
server_key: "0c9eea0257f87a866274bb8ea2f6b86b"
server_key_iv: "79c0e3b63c1275384138a57ddc47fca3"
server_key_version: "encryptionkey@0"
I20260926 03:06:01.849565 1724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:06:01.853796 1739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:01.855600 1724 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260926 03:06:01.855808 1724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "b871109b89de4856a53da7e7cc49af2b"
format_stamp: "Formatted at 2026-09-26 03:06:01 on dist-test-slave-hmbw"
server_key: "0c9eea0257f87a866274bb8ea2f6b86b"
server_key_iv: "79c0e3b63c1275384138a57ddc47fca3"
server_key_version: "encryptionkey@0"
I20260926 03:06:01.856034 1724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:01.882264 1724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:01.886828 1724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:01.887094 1724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:01.889845 1724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:01.889940 1724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:01.890061 1724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:01.890118 1724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:06:01 dist-test-slave-hmbw krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:01.916198 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:01.890606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43319 (local address 127.0.97.62:39621)
0926 03:06:01.890811 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:01.890817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:01.892368 (+ 1551us) server_negotiation.cc:407] Connection header received
0926 03:06:01.895241 (+ 2873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:01.895248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:01.895345 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:01.895464 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:01.897858 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:01.898840 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:01.901860 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:01.902556 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:01.902691 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:01.908474 (+ 5783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:01.908483 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:01.908488 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:01.908532 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:01.911962 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:01.913221 (+ 1259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:01.913228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:01.913232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:01.913399 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:01.913912 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:01.913919 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:01.913922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:01.914321 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:01.914466 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:01.915599 (+ 1133us) server_negotiation.cc:299] Negotiation successful
0926 03:06:01.915848 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:06:01.917407 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:01.891270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39621 (local address 127.0.97.2:43319)
0926 03:06:01.892148 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:01.892216 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:01.894834 (+ 2618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:01.895846 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:01.895860 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:01.896672 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:01.897588 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:01.897624 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:01.899066 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:01.899071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:01.901592 (+ 2521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:01.901615 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:01.903116 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:01.903134 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:01.903486 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:01.904347 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:01.904383 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:01.908062 (+ 3679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:01.912246 (+ 4184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:01.912270 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:01.912297 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:01.912995 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:01.913541 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:01.913544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:01.913547 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:01.913780 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:01.914457 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:01.914467 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:01.915134 (+ 667us) client_negotiation.cc:769] Sending connection context
0926 03:06:01.915454 (+ 320us) client_negotiation.cc:240] Negotiation successful
0926 03:06:01.916047 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":342,"threads_started":1}
I20260926 03:06:01.939997 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:01.925770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37599 (local address 127.0.97.62:39621)
0926 03:06:01.925981 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:01.925989 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:01.926585 (+ 596us) server_negotiation.cc:407] Connection header received
0926 03:06:01.926828 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:01.926834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:01.926905 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:01.926986 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:01.928169 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:01.929030 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:01.930332 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:01.930812 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:01.930960 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:01.933758 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:01.933766 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:01.933771 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:01.933814 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:01.936708 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:01.937746 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:01.937752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:01.937754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:01.937897 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:01.938432 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:01.938436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:01.938438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:01.938876 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:01.939022 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:01.939476 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 03:06:01.939690 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:06:01.940351 1833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:01.925741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39621 (local address 127.0.97.2:37599)
0926 03:06:01.926452 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:01.926472 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:01.926695 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:01.927034 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:01.927039 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:01.927601 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:01.928015 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:01.928028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:01.929164 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:01.929169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:01.930193 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:01.930205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:01.931018 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:01.931025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:01.931215 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:01.932003 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:01.932093 (+ 90us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:01.933511 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:01.936876 (+ 3365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:01.936883 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:01.936888 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:01.937521 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:01.938066 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:01.938073 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:01.938078 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:01.938284 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:01.939007 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:01.939012 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:01.939232 (+ 220us) client_negotiation.cc:769] Sending connection context
0926 03:06:01.939366 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 03:06:01.939895 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":403,"threads_started":1}
I20260926 03:06:01.941620 1858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37359 every 8 connection(s)
I20260926 03:06:01.941650 1724 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37359
I20260926 03:06:01.943347 1724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:06:01.943993 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 1724
I20260926 03:06:01.944192 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:06:01.944664 388 external_mini_cluster.cc:1428] Setting key 26b4c0287dd250ac485e91a488dc9241
I20260926 03:06:01.946141 1859 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39621
I20260926 03:06:01.946382 1859 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:01.947126 1859 heartbeater.cc:511] Master 127.0.97.62:39621 requested a full tablet report, sending...
I20260926 03:06:01.948904 1512 ts_manager.cc:194] Registered new tserver with Master: b871109b89de4856a53da7e7cc49af2b (127.0.97.2:37359)
I20260926 03:06:01.949796 1512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43319
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:06:01.987592 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39621
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:02.344223 1863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:02.344511 1863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:02.344609 1863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:02.352100 1863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:02.352228 1863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:02.352310 1863 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:06:02.352387 1863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:06:02.362792 1863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:02.364053 1863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:02.365384 1863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:02.380009 1868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:02.380264 1869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:02.380393 1871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:02.381605 1863 server_base.cc:1034] running on GCE node
I20260926 03:06:02.382297 1863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:02.383441 1863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:02.384682 1863 hybrid_clock.cc:648] HybridClock initialized: now 1790391962384615 us; error 43 us; skew 500 ppm
I20260926 03:06:02.384976 1863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:02 dist-test-slave-hmbw krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:02.388252 1863 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:06:02.389777 1863 webserver.cc:460] Webserver started at http://127.0.97.3:43523/ using document root <none> and password file <none>
I20260926 03:06:02.390458 1863 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:02.390590 1863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:02.390890 1863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:02.393846 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-2/data/instance:
uuid: "f5cd0c81921d44939e7fb31c9a125afe"
format_stamp: "Formatted at 2026-09-26 03:06:02 on dist-test-slave-hmbw"
server_key: "f45f036cc7b962636cbd83f6b4034fb8"
server_key_iv: "751094bcf6679cd70640d0412e991b58"
server_key_version: "encryptionkey@0"
I20260926 03:06:02.394721 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f5cd0c81921d44939e7fb31c9a125afe"
format_stamp: "Formatted at 2026-09-26 03:06:02 on dist-test-slave-hmbw"
server_key: "f45f036cc7b962636cbd83f6b4034fb8"
server_key_iv: "751094bcf6679cd70640d0412e991b58"
server_key_version: "encryptionkey@0"
I20260926 03:06:02.400074 1863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 03:06:02.403993 1878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:02.405797 1863 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 03:06:02.405988 1863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "f5cd0c81921d44939e7fb31c9a125afe"
format_stamp: "Formatted at 2026-09-26 03:06:02 on dist-test-slave-hmbw"
server_key: "f45f036cc7b962636cbd83f6b4034fb8"
server_key_iv: "751094bcf6679cd70640d0412e991b58"
server_key_version: "encryptionkey@0"
I20260926 03:06:02.406214 1863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:02.430781 1720 heartbeater.cc:503] Master 127.0.97.62:39621 was elected leader, sending a full tablet report...
I20260926 03:06:02.442307 1863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:02.446686 1863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:02.446951 1863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:02.449489 1863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:02.449580 1863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:02.449702 1863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:02.449764 1863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:06:02 dist-test-slave-hmbw krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:02.475317 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:02.450198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47211 (local address 127.0.97.62:39621)
0926 03:06:02.450645 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:02.450652 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:02.451928 (+ 1276us) server_negotiation.cc:407] Connection header received
0926 03:06:02.454357 (+ 2429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:02.454363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:02.454454 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:02.454580 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:02.457185 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.458213 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.461301 (+ 3088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.461959 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.462110 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:02.467362 (+ 5252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:02.467368 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:02.467372 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:02.467410 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:02.470704 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:02.471900 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:02.471906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:02.471910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:02.472067 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:02.472708 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:02.472715 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:02.472718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:02.473070 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:02.473223 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:02.474711 (+ 1488us) server_negotiation.cc:299] Negotiation successful
0926 03:06:02.474926 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":185,"threads_started":1}
I20260926 03:06:02.476159 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:02.450891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39621 (local address 127.0.97.3:47211)
0926 03:06:02.451740 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:02.451794 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:02.454018 (+ 2224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:02.454934 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:02.454946 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:02.455851 (+ 905us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:02.456937 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:02.456974 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.458413 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.458419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:02.460987 (+ 2568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:02.461011 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.462247 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.462257 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:02.462472 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:02.463279 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:02.463315 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:02.467039 (+ 3724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:02.470883 (+ 3844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:02.470895 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:02.470925 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:02.471695 (+ 770us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:02.472224 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:02.472229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:02.472232 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:02.472541 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:02.473307 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:02.473320 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:02.473854 (+ 534us) client_negotiation.cc:769] Sending connection context
0926 03:06:02.474167 (+ 313us) client_negotiation.cc:240] Negotiation successful
0926 03:06:02.474818 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":351,"threads_started":1}
I20260926 03:06:02.496977 1863 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44393
I20260926 03:06:02.497068 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:02.482310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60441 (local address 127.0.97.62:39621)
0926 03:06:02.482539 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:02.482545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:02.483573 (+ 1028us) server_negotiation.cc:407] Connection header received
0926 03:06:02.483771 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:02.483777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:02.483871 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:02.483960 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:02.485003 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.485896 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.487895 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.488480 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.488624 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:02.491255 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:02.491261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:02.491264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:02.491297 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:02.493853 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:02.494725 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:02.494732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:02.494736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:02.494928 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:02.495370 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:02.495383 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:02.495386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:02.495762 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:02.495922 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:02.496454 (+ 532us) server_negotiation.cc:299] Negotiation successful
0926 03:06:02.496725 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 03:06:02.497038 1998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44393 every 8 connection(s)
I20260926 03:06:02.497328 1971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:02.482671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39621 (local address 127.0.97.3:60441)
0926 03:06:02.483446 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:02.483471 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:02.483642 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:02.484009 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:02.484014 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:02.484535 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:02.484844 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:02.484855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.486084 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.486090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:02.487719 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:02.487747 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.488727 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.488738 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:02.488961 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:02.490028 (+ 1067us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:02.490051 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:02.491030 (+ 979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:02.494022 (+ 2992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:02.494028 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:02.494033 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:02.494481 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:02.495083 (+ 602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:02.495087 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:02.495090 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:02.495238 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:02.495915 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:02.495921 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:02.496165 (+ 244us) client_negotiation.cc:769] Sending connection context
0926 03:06:02.496307 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 03:06:02.496770 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":369,"threads_started":1}
I20260926 03:06:02.498703 1863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:06:02.499013 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 1863
I20260926 03:06:02.499191 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:06:02.499701 388 external_mini_cluster.cc:1428] Setting key de752946ed9348494697a9dc9e296592
I20260926 03:06:02.502592 1999 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39621
I20260926 03:06:02.502839 1999 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:02.503523 1999 heartbeater.cc:511] Master 127.0.97.62:39621 requested a full tablet report, sending...
I20260926 03:06:02.504863 1513 ts_manager.cc:194] Registered new tserver with Master: f5cd0c81921d44939e7fb31c9a125afe (127.0.97.3:44393)
I20260926 03:06:02.505681 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47211
I20260926 03:06:02.516538 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 03:06:02 dist-test-slave-hmbw krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391962, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 03:06:02 dist-test-slave-hmbw krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391962, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:02.561367 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:02.547028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44360 (local address 127.0.97.62:39621)
0926 03:06:02.547191 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:02.547195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:02.547456 (+ 261us) server_negotiation.cc:407] Connection header received
0926 03:06:02.547580 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:02.547585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:02.547657 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:02.547749 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:02.549142 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.549927 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.551297 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.551708 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.551840 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:02.555864 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:02.555870 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:02.555874 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:02.555918 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:02.558613 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:02.559359 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:02.559365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:02.559368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:02.559538 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:02.559932 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:02.559937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:02.559939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:02.560233 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:02.560473 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:02.560870 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 03:06:02.561045 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 03:06:02.566075 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:06:02.572974 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:06:02.611382 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:02.586250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51672 (local address 127.0.97.3:44393)
0926 03:06:02.600334 (+ 14084us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:02.600345 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:02.601894 (+ 1549us) server_negotiation.cc:407] Connection header received
0926 03:06:02.602043 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:02.602052 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:02.602526 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:02.602740 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:02.604123 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.605504 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.608102 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.609517 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.609653 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:02.609973 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:02.610830 (+ 857us) server_negotiation.cc:299] Negotiation successful
0926 03:06:02.611043 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13865,"thread_start_us":16441,"threads_started":1}
I20260926 03:06:02.611361 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:02.585782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49548 (local address 127.0.97.1:41143)
0926 03:06:02.602150 (+ 16368us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:02.602159 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:02.602185 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:06:02.602282 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:02.602289 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:02.602526 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:02.602736 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:02.603855 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.605365 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.607962 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.609684 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.609804 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:02.610007 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:02.610913 (+ 906us) server_negotiation.cc:299] Negotiation successful
0926 03:06:02.611095 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16208,"thread_start_us":18040,"threads_started":1}
I20260926 03:06:02.611400 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:02.585730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44393 (local address 127.0.0.1:51672)
0926 03:06:02.601554 (+ 15824us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:02.601577 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:02.601753 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:02.603365 (+ 1612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:02.603370 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:02.603404 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:02.603898 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:02.603911 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.605650 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.605654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:02.607926 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:02.607946 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.610132 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.610138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:02.610429 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:02.610583 (+ 154us) client_negotiation.cc:769] Sending connection context
0926 03:06:02.610728 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 03:06:02.610992 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15652,"mutex_wait_us":219,"thread_start_us":12107,"threads_started":1}
I20260926 03:06:02.611649 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:02.599187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58442 (local address 127.0.97.2:37359)
0926 03:06:02.603324 (+ 4137us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:02.603332 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:02.604065 (+ 733us) server_negotiation.cc:407] Connection header received
0926 03:06:02.604169 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:02.604177 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:02.604472 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:02.604667 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:02.605530 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.606959 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.609140 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.610360 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.610483 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:02.610711 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:02.611211 (+ 500us) server_negotiation.cc:299] Negotiation successful
0926 03:06:02.611375 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3994,"thread_start_us":3877,"threads_started":1}
I20260926 03:06:02.612115 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:02.585648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41143 (local address 127.0.0.1:49548)
0926 03:06:02.598305 (+ 12657us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:02.598352 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:02.598604 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:02.603034 (+ 4430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:02.603044 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:02.603093 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:02.603662 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:02.603681 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.605520 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.605524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:02.607792 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:02.607805 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.610183 (+ 2378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.610189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:02.610447 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:02.610593 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 03:06:02.610724 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 03:06:02.611755 (+ 1031us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12351,"mutex_wait_us":973,"thread_start_us":15590,"threads_started":1}
I20260926 03:06:02.612304 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:02.598967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37359 (local address 127.0.0.1:58442)
0926 03:06:02.603844 (+ 4877us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:02.603867 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:02.603992 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:02.604865 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:02.604870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:02.604895 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:02.605324 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:02.605336 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.607108 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.607113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:02.608956 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:02.608969 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.610600 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.610607 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:02.610801 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:02.610906 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 03:06:02.610990 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 03:06:02.611911 (+ 921us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4720,"mutex_wait_us":17,"thread_start_us":4542,"threads_started":1}
I20260926 03:06:02.615727 1650 tablet_service.cc:1467] Processing CreateTablet for tablet ca6844c8d0cb4cb0828051db4de97e30 (DEFAULT_TABLE table=test-table [id=68d6d401440f40e587901eee6559ee9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:02.615727 1929 tablet_service.cc:1467] Processing CreateTablet for tablet ca6844c8d0cb4cb0828051db4de97e30 (DEFAULT_TABLE table=test-table [id=68d6d401440f40e587901eee6559ee9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:02.616868 1650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca6844c8d0cb4cb0828051db4de97e30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:02.616880 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca6844c8d0cb4cb0828051db4de97e30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:02.621102 1789 tablet_service.cc:1467] Processing CreateTablet for tablet ca6844c8d0cb4cb0828051db4de97e30 (DEFAULT_TABLE table=test-table [id=68d6d401440f40e587901eee6559ee9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:02.622614 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca6844c8d0cb4cb0828051db4de97e30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:02.629136 2018 tablet_bootstrap.cc:492] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe: Bootstrap starting.
I20260926 03:06:02.632117 2018 tablet_bootstrap.cc:654] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:02.632993 2019 tablet_bootstrap.cc:492] T ca6844c8d0cb4cb0828051db4de97e30 P bcc8fba247f64f4c8d6d4e2aaac951f9: Bootstrap starting.
I20260926 03:06:02.633481 2018 log.cc:824] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:02.636615 2019 tablet_bootstrap.cc:654] T ca6844c8d0cb4cb0828051db4de97e30 P bcc8fba247f64f4c8d6d4e2aaac951f9: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:02.636902 2018 tablet_bootstrap.cc:492] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe: No bootstrap required, opened a new log
I20260926 03:06:02.637197 2018 ts_tablet_manager.cc:1397] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 03:06:02.637678 2019 log.cc:824] T ca6844c8d0cb4cb0828051db4de97e30 P bcc8fba247f64f4c8d6d4e2aaac951f9: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:02.638540 2021 tablet_bootstrap.cc:492] T ca6844c8d0cb4cb0828051db4de97e30 P b871109b89de4856a53da7e7cc49af2b: Bootstrap starting.
I20260926 03:06:02.642033 2021 tablet_bootstrap.cc:654] T ca6844c8d0cb4cb0828051db4de97e30 P b871109b89de4856a53da7e7cc49af2b: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:02.642297 2019 tablet_bootstrap.cc:492] T ca6844c8d0cb4cb0828051db4de97e30 P bcc8fba247f64f4c8d6d4e2aaac951f9: No bootstrap required, opened a new log
I20260926 03:06:02.642619 2019 ts_tablet_manager.cc:1397] T ca6844c8d0cb4cb0828051db4de97e30 P bcc8fba247f64f4c8d6d4e2aaac951f9: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20260926 03:06:02.643285 2021 log.cc:824] T ca6844c8d0cb4cb0828051db4de97e30 P b871109b89de4856a53da7e7cc49af2b: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:02.645650 2018 raft_consensus.cc:348] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } }
I20260926 03:06:02.646009 2018 raft_consensus.cc:374] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:02.646113 2018 raft_consensus.cc:729] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5cd0c81921d44939e7fb31c9a125afe, State: Initialized, Role: FOLLOWER
I20260926 03:06:02.647065 2018 consensus_queue.cc:260] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } }
I20260926 03:06:02.648170 1999 heartbeater.cc:503] Master 127.0.97.62:39621 was elected leader, sending a full tablet report...
I20260926 03:06:02.649471 2021 tablet_bootstrap.cc:492] T ca6844c8d0cb4cb0828051db4de97e30 P b871109b89de4856a53da7e7cc49af2b: No bootstrap required, opened a new log
I20260926 03:06:02.650017 2018 ts_tablet_manager.cc:1428] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260926 03:06:02.650372 2021 ts_tablet_manager.cc:1397] T ca6844c8d0cb4cb0828051db4de97e30 P b871109b89de4856a53da7e7cc49af2b: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260926 03:06:02.652740 2019 raft_consensus.cc:348] T ca6844c8d0cb4cb0828051db4de97e30 P bcc8fba247f64f4c8d6d4e2aaac951f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } }
I20260926 03:06:02.653112 2019 raft_consensus.cc:374] T ca6844c8d0cb4cb0828051db4de97e30 P bcc8fba247f64f4c8d6d4e2aaac951f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:02.653205 2019 raft_consensus.cc:729] T ca6844c8d0cb4cb0828051db4de97e30 P bcc8fba247f64f4c8d6d4e2aaac951f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcc8fba247f64f4c8d6d4e2aaac951f9, State: Initialized, Role: FOLLOWER
I20260926 03:06:02.653811 2019 consensus_queue.cc:260] T ca6844c8d0cb4cb0828051db4de97e30 P bcc8fba247f64f4c8d6d4e2aaac951f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } }
I20260926 03:06:02.656365 2019 ts_tablet_manager.cc:1428] T ca6844c8d0cb4cb0828051db4de97e30 P bcc8fba247f64f4c8d6d4e2aaac951f9: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260926 03:06:02.659848 2021 raft_consensus.cc:348] T ca6844c8d0cb4cb0828051db4de97e30 P b871109b89de4856a53da7e7cc49af2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } }
I20260926 03:06:02.660379 2021 raft_consensus.cc:374] T ca6844c8d0cb4cb0828051db4de97e30 P b871109b89de4856a53da7e7cc49af2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:02.660502 2021 raft_consensus.cc:729] T ca6844c8d0cb4cb0828051db4de97e30 P b871109b89de4856a53da7e7cc49af2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b871109b89de4856a53da7e7cc49af2b, State: Initialized, Role: FOLLOWER
I20260926 03:06:02.661226 2021 consensus_queue.cc:260] T ca6844c8d0cb4cb0828051db4de97e30 P b871109b89de4856a53da7e7cc49af2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } }
I20260926 03:06:02.662281 1859 heartbeater.cc:503] Master 127.0.97.62:39621 was elected leader, sending a full tablet report...
I20260926 03:06:02.662995 2021 ts_tablet_manager.cc:1428] T ca6844c8d0cb4cb0828051db4de97e30 P b871109b89de4856a53da7e7cc49af2b: Time spent starting tablet: real 0.013s user 0.005s sys 0.008s
W20260926 03:06:02.672086 1721 tablet.cc:2369] T ca6844c8d0cb4cb0828051db4de97e30 P bcc8fba247f64f4c8d6d4e2aaac951f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:06:02.696738 1860 tablet.cc:2369] T ca6844c8d0cb4cb0828051db4de97e30 P b871109b89de4856a53da7e7cc49af2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:06:02.752681 2000 tablet.cc:2369] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:06:02.782892 2024 raft_consensus.cc:482] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:06:02.783231 2024 raft_consensus.cc:504] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } }
I20260926 03:06:02.785284 2024 leader_election.cc:288] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcc8fba247f64f4c8d6d4e2aaac951f9 (127.0.97.1:41143), b871109b89de4856a53da7e7cc49af2b (127.0.97.2:37359)
I20260926 03:06:02.788992 2026 raft_consensus.cc:482] T ca6844c8d0cb4cb0828051db4de97e30 P b871109b89de4856a53da7e7cc49af2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:06:02.789343 2026 raft_consensus.cc:504] T ca6844c8d0cb4cb0828051db4de97e30 P b871109b89de4856a53da7e7cc49af2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } }
I20260926 03:06:02.791494 2026 leader_election.cc:288] T ca6844c8d0cb4cb0828051db4de97e30 P b871109b89de4856a53da7e7cc49af2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcc8fba247f64f4c8d6d4e2aaac951f9 (127.0.97.1:41143), f5cd0c81921d44939e7fb31c9a125afe (127.0.97.3:44393)
I20260926 03:06:02.793370 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:02.785668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41143 (local address 127.0.97.3:37855)
0926 03:06:02.786072 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:02.786093 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:02.786232 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:02.786808 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:02.786813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:02.786844 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:02.787057 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:02.787071 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.788329 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.788333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:02.790601 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:02.790618 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.792407 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.792413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:02.792813 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:02.792849 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 03:06:02.792942 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 03:06:02.793085 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236}
I20260926 03:06:02.794063 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:02.786120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37855 (local address 127.0.97.1:41143)
0926 03:06:02.786302 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:02.786308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:02.786328 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:06:02.786403 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:02.786408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:02.786488 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:02.786598 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:02.787223 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.788196 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.790851 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.792040 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.792177 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:02.792294 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:02.793682 (+ 1388us) server_negotiation.cc:299] Negotiation successful
0926 03:06:02.793824 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:06:02.794922 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:02.785735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43759 (local address 127.0.97.2:37359)
0926 03:06:02.786528 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:02.786535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:02.787115 (+ 580us) server_negotiation.cc:407] Connection header received
0926 03:06:02.787232 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:02.787237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:02.787309 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:02.787411 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:02.788060 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.789788 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.792298 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.793547 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.793662 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:02.793775 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:02.794388 (+ 613us) server_negotiation.cc:299] Negotiation successful
0926 03:06:02.794583 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638}
I20260926 03:06:02.795711 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:02.785869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37359 (local address 127.0.97.3:43759)
0926 03:06:02.786995 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:02.787016 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:02.787132 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:02.787533 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:02.787538 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:02.787565 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:02.787891 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:02.787902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.790001 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.790007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:02.792100 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:02.792118 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.793995 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.794002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:02.794206 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:02.794240 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 03:06:02.795173 (+ 933us) client_negotiation.cc:240] Negotiation successful
0926 03:06:02.795354 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":993,"thread_start_us":787,"threads_started":1}
I20260926 03:06:02.795825 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6844c8d0cb4cb0828051db4de97e30" candidate_uuid: "f5cd0c81921d44939e7fb31c9a125afe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" is_pre_election: true
I20260926 03:06:02.796468 1670 raft_consensus.cc:2454] T ca6844c8d0cb4cb0828051db4de97e30 P bcc8fba247f64f4c8d6d4e2aaac951f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5cd0c81921d44939e7fb31c9a125afe in term 0.
I20260926 03:06:02.797511 1879 leader_election.cc:302] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcc8fba247f64f4c8d6d4e2aaac951f9, f5cd0c81921d44939e7fb31c9a125afe; no voters:
I20260926 03:06:02.797995 2024 raft_consensus.cc:2781] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:06:02.798137 2024 raft_consensus.cc:482] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:06:02.798226 2024 raft_consensus.cc:3037] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:02.801342 2024 raft_consensus.cc:504] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } }
I20260926 03:06:02.803139 2024 leader_election.cc:288] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe [CANDIDATE]: Term 1 election: Requested vote from peers bcc8fba247f64f4c8d6d4e2aaac951f9 (127.0.97.1:41143), b871109b89de4856a53da7e7cc49af2b (127.0.97.2:37359)
I20260926 03:06:02.803678 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:02.792523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60915 (local address 127.0.97.3:44393)
0926 03:06:02.792712 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:02.792718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:02.795701 (+ 2983us) server_negotiation.cc:407] Connection header received
0926 03:06:02.796068 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:02.796073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:02.796147 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:02.796208 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:02.797802 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.798878 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.801319 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.802462 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.802576 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:02.802679 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:02.803290 (+ 611us) server_negotiation.cc:299] Negotiation successful
0926 03:06:02.803429 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 03:06:02.803793 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6844c8d0cb4cb0828051db4de97e30" candidate_uuid: "f5cd0c81921d44939e7fb31c9a125afe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b871109b89de4856a53da7e7cc49af2b" is_pre_election: true
I20260926 03:06:02.804201 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6844c8d0cb4cb0828051db4de97e30" candidate_uuid: "f5cd0c81921d44939e7fb31c9a125afe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b871109b89de4856a53da7e7cc49af2b"
I20260926 03:06:02.804340 1809 raft_consensus.cc:2454] T ca6844c8d0cb4cb0828051db4de97e30 P b871109b89de4856a53da7e7cc49af2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5cd0c81921d44939e7fb31c9a125afe in term 0.
I20260926 03:06:02.804473 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6844c8d0cb4cb0828051db4de97e30" candidate_uuid: "f5cd0c81921d44939e7fb31c9a125afe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9"
I20260926 03:06:02.804688 1670 raft_consensus.cc:3037] T ca6844c8d0cb4cb0828051db4de97e30 P bcc8fba247f64f4c8d6d4e2aaac951f9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:02.805406 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:02.792083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44393 (local address 127.0.97.2:60915)
0926 03:06:02.795598 (+ 3515us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:02.795621 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:02.795901 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:02.796406 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:02.796410 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:02.796437 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:02.797616 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:02.797633 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.799055 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.799061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:02.801113 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:02.801128 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.802893 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.802899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:02.803096 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:02.803129 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:06:02.803838 (+ 709us) client_negotiation.cc:240] Negotiation successful
0926 03:06:02.803995 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3399,"threads_started":1}
I20260926 03:06:02.807118 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6844c8d0cb4cb0828051db4de97e30" candidate_uuid: "b871109b89de4856a53da7e7cc49af2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5cd0c81921d44939e7fb31c9a125afe" is_pre_election: true
I20260926 03:06:02.808019 1949 raft_consensus.cc:2379] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b871109b89de4856a53da7e7cc49af2b in current term 1: Already voted for candidate f5cd0c81921d44939e7fb31c9a125afe in this term.
I20260926 03:06:02.808703 1670 raft_consensus.cc:2454] T ca6844c8d0cb4cb0828051db4de97e30 P bcc8fba247f64f4c8d6d4e2aaac951f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5cd0c81921d44939e7fb31c9a125afe in term 1.
I20260926 03:06:02.809060 1879 leader_election.cc:302] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bcc8fba247f64f4c8d6d4e2aaac951f9, f5cd0c81921d44939e7fb31c9a125afe; no voters: b871109b89de4856a53da7e7cc49af2b
I20260926 03:06:02.809383 2024 raft_consensus.cc:2781] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:02.809914 2024 raft_consensus.cc:686] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe [term 1 LEADER]: Becoming Leader. State: Replica: f5cd0c81921d44939e7fb31c9a125afe, State: Running, Role: LEADER
I20260926 03:06:02.810953 2024 consensus_queue.cc:237] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } }
I20260926 03:06:02.812599 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:02.793067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41143 (local address 127.0.97.2:45725)
0926 03:06:02.805707 (+ 12640us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:02.805728 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:02.805871 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:02.806395 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:02.806400 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:02.806429 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:02.806635 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:02.806647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.808365 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.808370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:02.809913 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:02.809932 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.811964 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.811972 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:02.812129 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:02.812158 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 03:06:02.812254 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 03:06:02.812378 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12503,"thread_start_us":9563,"threads_started":1}
I20260926 03:06:02.812701 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:02.793486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45725 (local address 127.0.97.1:41143)
0926 03:06:02.794653 (+ 1167us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:02.794667 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:02.805803 (+ 11136us) server_negotiation.cc:407] Connection header received
0926 03:06:02.806045 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:02.806051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:02.806140 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:02.806238 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:02.806819 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.808227 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.810086 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.811640 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.811929 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:02.812038 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:02.812292 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 03:06:02.812428 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":967,"thread_start_us":9155,"threads_started":1}
I20260926 03:06:02.813570 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6844c8d0cb4cb0828051db4de97e30" candidate_uuid: "b871109b89de4856a53da7e7cc49af2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" is_pre_election: true
I20260926 03:06:02.813941 1670 raft_consensus.cc:2379] T ca6844c8d0cb4cb0828051db4de97e30 P bcc8fba247f64f4c8d6d4e2aaac951f9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b871109b89de4856a53da7e7cc49af2b in current term 1: Already voted for candidate f5cd0c81921d44939e7fb31c9a125afe in this term.
I20260926 03:06:02.814644 1740 leader_election.cc:302] T ca6844c8d0cb4cb0828051db4de97e30 P b871109b89de4856a53da7e7cc49af2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b871109b89de4856a53da7e7cc49af2b; no voters: bcc8fba247f64f4c8d6d4e2aaac951f9, f5cd0c81921d44939e7fb31c9a125afe
I20260926 03:06:02.815052 2026 raft_consensus.cc:3037] T ca6844c8d0cb4cb0828051db4de97e30 P b871109b89de4856a53da7e7cc49af2b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:02.816421 1513 catalog_manager.cc:5495] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe reported cstate change: term changed from 0 to 1, leader changed from <none> to f5cd0c81921d44939e7fb31c9a125afe (127.0.97.3). New cstate: current_term: 1 leader_uuid: "f5cd0c81921d44939e7fb31c9a125afe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } health_report { overall_health: UNKNOWN } } }
I20260926 03:06:02.817262 2026 raft_consensus.cc:2726] T ca6844c8d0cb4cb0828051db4de97e30 P b871109b89de4856a53da7e7cc49af2b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 03:06:02.864275 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:02.857818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51680 (local address 127.0.97.3:44393)
0926 03:06:02.858081 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:02.858088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:02.858657 (+ 569us) server_negotiation.cc:407] Connection header received
0926 03:06:02.858954 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:02.858958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:02.859055 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:02.859215 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:06:02.859622 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.860420 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.861940 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:02.862319 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:02.862422 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:02.863106 (+ 684us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:06:02.863692 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:06:02.863802 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:02.863977 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 03:06:02.864049 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 03:06:02.873011 1808 raft_consensus.cc:1261] T ca6844c8d0cb4cb0828051db4de97e30 P b871109b89de4856a53da7e7cc49af2b [term 1 FOLLOWER]: Refusing update from remote peer f5cd0c81921d44939e7fb31c9a125afe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:02.873011 1670 raft_consensus.cc:1261] T ca6844c8d0cb4cb0828051db4de97e30 P bcc8fba247f64f4c8d6d4e2aaac951f9 [term 1 FOLLOWER]: Refusing update from remote peer f5cd0c81921d44939e7fb31c9a125afe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:02.874045 2031 consensus_queue.cc:1035] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:02.874466 2024 consensus_queue.cc:1035] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:02.886641 2035 mvcc.cc:205] Tried to move back new op lower bound from 7333445479915663360 to 7333445479678914560. Current Snapshot: MvccSnapshot[applied={T|T < 7333445479915663360}]
I20260926 03:06:02.887989 2034 mvcc.cc:205] Tried to move back new op lower bound from 7333445479915663360 to 7333445479678914560. Current Snapshot: MvccSnapshot[applied={T|T < 7333445479915663360}]
I20260926 03:06:02.910926 1513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:06:02.911234 1513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:06:02.915230 1513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 745f1df3f2994915bcd8fe7b399f03ae: Sending DeleteTablet for 3 replicas of tablet ca6844c8d0cb4cb0828051db4de97e30
I20260926 03:06:02.916570 1650 tablet_service.cc:1514] Processing DeleteTablet for tablet ca6844c8d0cb4cb0828051db4de97e30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49548
I20260926 03:06:02.916651 1789 tablet_service.cc:1514] Processing DeleteTablet for tablet ca6844c8d0cb4cb0828051db4de97e30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58442
I20260926 03:06:02.917672 1929 tablet_service.cc:1514] Processing DeleteTablet for tablet ca6844c8d0cb4cb0828051db4de97e30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51672
I20260926 03:06:02.918123 1512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:06:02.918887 1512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:06:02.922325 2043 tablet_replica.cc:330] T ca6844c8d0cb4cb0828051db4de97e30 P b871109b89de4856a53da7e7cc49af2b: stopping tablet replica
I20260926 03:06:02.923559 2044 tablet_replica.cc:330] T ca6844c8d0cb4cb0828051db4de97e30 P bcc8fba247f64f4c8d6d4e2aaac951f9: stopping tablet replica
I20260926 03:06:02.924136 2044 raft_consensus.cc:2229] T ca6844c8d0cb4cb0828051db4de97e30 P bcc8fba247f64f4c8d6d4e2aaac951f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:06:02.925319 2044 raft_consensus.cc:2258] T ca6844c8d0cb4cb0828051db4de97e30 P bcc8fba247f64f4c8d6d4e2aaac951f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:02.925922 2043 raft_consensus.cc:2229] T ca6844c8d0cb4cb0828051db4de97e30 P b871109b89de4856a53da7e7cc49af2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:06:02.927529 2043 raft_consensus.cc:2258] T ca6844c8d0cb4cb0828051db4de97e30 P b871109b89de4856a53da7e7cc49af2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:02.930085 2043 ts_tablet_manager.cc:1905] T ca6844c8d0cb4cb0828051db4de97e30 P b871109b89de4856a53da7e7cc49af2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:02.930284 1649 tablet_service.cc:1467] Processing CreateTablet for tablet eaca44d2cdc249289ff451faf67216d4 (DEFAULT_TABLE table=test-table [id=fc3e18a5ee3f46b4a3bb54d4d7fb6027]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:02.930547 2045 tablet_replica.cc:330] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe: stopping tablet replica
I20260926 03:06:02.930815 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaca44d2cdc249289ff451faf67216d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:02.931018 2045 raft_consensus.cc:2229] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:06:02.931664 2045 raft_consensus.cc:2258] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:02.934849 1789 tablet_service.cc:1467] Processing CreateTablet for tablet eaca44d2cdc249289ff451faf67216d4 (DEFAULT_TABLE table=test-table [id=fc3e18a5ee3f46b4a3bb54d4d7fb6027]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:02.935804 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaca44d2cdc249289ff451faf67216d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:02.936043 1929 tablet_service.cc:1467] Processing CreateTablet for tablet eaca44d2cdc249289ff451faf67216d4 (DEFAULT_TABLE table=test-table [id=fc3e18a5ee3f46b4a3bb54d4d7fb6027]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:02.936509 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaca44d2cdc249289ff451faf67216d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:02.941293 2019 tablet_bootstrap.cc:492] T eaca44d2cdc249289ff451faf67216d4 P bcc8fba247f64f4c8d6d4e2aaac951f9: Bootstrap starting.
I20260926 03:06:02.941727 2044 ts_tablet_manager.cc:1905] T ca6844c8d0cb4cb0828051db4de97e30 P bcc8fba247f64f4c8d6d4e2aaac951f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:02.942934 2019 tablet_bootstrap.cc:654] T eaca44d2cdc249289ff451faf67216d4 P bcc8fba247f64f4c8d6d4e2aaac951f9: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:02.944867 2021 tablet_bootstrap.cc:492] T eaca44d2cdc249289ff451faf67216d4 P b871109b89de4856a53da7e7cc49af2b: Bootstrap starting.
I20260926 03:06:02.946671 2021 tablet_bootstrap.cc:654] T eaca44d2cdc249289ff451faf67216d4 P b871109b89de4856a53da7e7cc49af2b: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:02.948326 2043 ts_tablet_manager.cc:1918] T ca6844c8d0cb4cb0828051db4de97e30 P b871109b89de4856a53da7e7cc49af2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:06:02.948588 2043 log.cc:1196] T ca6844c8d0cb4cb0828051db4de97e30 P b871109b89de4856a53da7e7cc49af2b: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-1/wal/wals/ca6844c8d0cb4cb0828051db4de97e30
I20260926 03:06:02.949288 2043 ts_tablet_manager.cc:1939] T ca6844c8d0cb4cb0828051db4de97e30 P b871109b89de4856a53da7e7cc49af2b: Deleting consensus metadata
I20260926 03:06:02.951380 1498 catalog_manager.cc:4841] TS b871109b89de4856a53da7e7cc49af2b (127.0.97.2:37359): tablet ca6844c8d0cb4cb0828051db4de97e30 (table test-table [id=68d6d401440f40e587901eee6559ee9c]) successfully deleted
I20260926 03:06:02.953397 2044 ts_tablet_manager.cc:1918] T ca6844c8d0cb4cb0828051db4de97e30 P bcc8fba247f64f4c8d6d4e2aaac951f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:06:02.953696 2044 log.cc:1196] T ca6844c8d0cb4cb0828051db4de97e30 P bcc8fba247f64f4c8d6d4e2aaac951f9: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-0/wal/wals/ca6844c8d0cb4cb0828051db4de97e30
I20260926 03:06:02.954452 2044 ts_tablet_manager.cc:1939] T ca6844c8d0cb4cb0828051db4de97e30 P bcc8fba247f64f4c8d6d4e2aaac951f9: Deleting consensus metadata
I20260926 03:06:02.956614 1497 catalog_manager.cc:4841] TS bcc8fba247f64f4c8d6d4e2aaac951f9 (127.0.97.1:41143): tablet ca6844c8d0cb4cb0828051db4de97e30 (table test-table [id=68d6d401440f40e587901eee6559ee9c]) successfully deleted
I20260926 03:06:02.952504 2045 ts_tablet_manager.cc:1905] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:02.957906 2018 tablet_bootstrap.cc:492] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe: Bootstrap starting.
I20260926 03:06:02.960026 2018 tablet_bootstrap.cc:654] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:02.961925 2021 tablet_bootstrap.cc:492] T eaca44d2cdc249289ff451faf67216d4 P b871109b89de4856a53da7e7cc49af2b: No bootstrap required, opened a new log
I20260926 03:06:02.962102 2021 ts_tablet_manager.cc:1397] T eaca44d2cdc249289ff451faf67216d4 P b871109b89de4856a53da7e7cc49af2b: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.006s
I20260926 03:06:02.963054 2021 raft_consensus.cc:348] T eaca44d2cdc249289ff451faf67216d4 P b871109b89de4856a53da7e7cc49af2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } }
I20260926 03:06:02.963558 2021 raft_consensus.cc:374] T eaca44d2cdc249289ff451faf67216d4 P b871109b89de4856a53da7e7cc49af2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:02.963718 2021 raft_consensus.cc:729] T eaca44d2cdc249289ff451faf67216d4 P b871109b89de4856a53da7e7cc49af2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b871109b89de4856a53da7e7cc49af2b, State: Initialized, Role: FOLLOWER
I20260926 03:06:02.964109 2021 consensus_queue.cc:260] T eaca44d2cdc249289ff451faf67216d4 P b871109b89de4856a53da7e7cc49af2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } }
I20260926 03:06:02.968514 2021 ts_tablet_manager.cc:1428] T eaca44d2cdc249289ff451faf67216d4 P b871109b89de4856a53da7e7cc49af2b: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260926 03:06:02.980823 2018 tablet_bootstrap.cc:492] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe: No bootstrap required, opened a new log
I20260926 03:06:02.981061 2018 ts_tablet_manager.cc:1397] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20260926 03:06:02.982045 2018 raft_consensus.cc:348] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } }
I20260926 03:06:02.982329 2018 raft_consensus.cc:374] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:02.982425 2018 raft_consensus.cc:729] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5cd0c81921d44939e7fb31c9a125afe, State: Initialized, Role: FOLLOWER
I20260926 03:06:02.982805 2018 consensus_queue.cc:260] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } }
I20260926 03:06:02.983928 2018 ts_tablet_manager.cc:1428] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 03:06:02.985692 2019 tablet_bootstrap.cc:492] T eaca44d2cdc249289ff451faf67216d4 P bcc8fba247f64f4c8d6d4e2aaac951f9: No bootstrap required, opened a new log
I20260926 03:06:02.985899 2019 ts_tablet_manager.cc:1397] T eaca44d2cdc249289ff451faf67216d4 P bcc8fba247f64f4c8d6d4e2aaac951f9: Time spent bootstrapping tablet: real 0.045s user 0.016s sys 0.024s
I20260926 03:06:02.987432 2045 ts_tablet_manager.cc:1918] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:06:02.987699 2045 log.cc:1196] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-2/wal/wals/ca6844c8d0cb4cb0828051db4de97e30
I20260926 03:06:02.987567 2019 raft_consensus.cc:348] T eaca44d2cdc249289ff451faf67216d4 P bcc8fba247f64f4c8d6d4e2aaac951f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } }
I20260926 03:06:02.987824 2019 raft_consensus.cc:374] T eaca44d2cdc249289ff451faf67216d4 P bcc8fba247f64f4c8d6d4e2aaac951f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:02.987967 2019 raft_consensus.cc:729] T eaca44d2cdc249289ff451faf67216d4 P bcc8fba247f64f4c8d6d4e2aaac951f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcc8fba247f64f4c8d6d4e2aaac951f9, State: Initialized, Role: FOLLOWER
I20260926 03:06:02.988422 2045 ts_tablet_manager.cc:1939] T ca6844c8d0cb4cb0828051db4de97e30 P f5cd0c81921d44939e7fb31c9a125afe: Deleting consensus metadata
I20260926 03:06:02.988310 2019 consensus_queue.cc:260] T eaca44d2cdc249289ff451faf67216d4 P bcc8fba247f64f4c8d6d4e2aaac951f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } }
I20260926 03:06:02.988955 2019 ts_tablet_manager.cc:1428] T eaca44d2cdc249289ff451faf67216d4 P bcc8fba247f64f4c8d6d4e2aaac951f9: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260926 03:06:02.990074 1498 catalog_manager.cc:4841] TS f5cd0c81921d44939e7fb31c9a125afe (127.0.97.3:44393): tablet ca6844c8d0cb4cb0828051db4de97e30 (table test-table [id=68d6d401440f40e587901eee6559ee9c]) successfully deleted
I20260926 03:06:03.031980 2031 raft_consensus.cc:482] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:06:03.032246 2031 raft_consensus.cc:504] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } }
I20260926 03:06:03.032998 2031 leader_election.cc:288] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcc8fba247f64f4c8d6d4e2aaac951f9 (127.0.97.1:41143), b871109b89de4856a53da7e7cc49af2b (127.0.97.2:37359)
I20260926 03:06:03.033599 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaca44d2cdc249289ff451faf67216d4" candidate_uuid: "f5cd0c81921d44939e7fb31c9a125afe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" is_pre_election: true
I20260926 03:06:03.033676 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaca44d2cdc249289ff451faf67216d4" candidate_uuid: "f5cd0c81921d44939e7fb31c9a125afe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b871109b89de4856a53da7e7cc49af2b" is_pre_election: true
I20260926 03:06:03.033865 1670 raft_consensus.cc:2454] T eaca44d2cdc249289ff451faf67216d4 P bcc8fba247f64f4c8d6d4e2aaac951f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5cd0c81921d44939e7fb31c9a125afe in term 0.
I20260926 03:06:03.033932 1808 raft_consensus.cc:2454] T eaca44d2cdc249289ff451faf67216d4 P b871109b89de4856a53da7e7cc49af2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5cd0c81921d44939e7fb31c9a125afe in term 0.
I20260926 03:06:03.034304 1879 leader_election.cc:302] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcc8fba247f64f4c8d6d4e2aaac951f9, f5cd0c81921d44939e7fb31c9a125afe; no voters:
I20260926 03:06:03.034631 2031 raft_consensus.cc:2781] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:06:03.034758 2031 raft_consensus.cc:482] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:06:03.034871 2031 raft_consensus.cc:3037] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:03.038009 2031 raft_consensus.cc:504] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } }
I20260926 03:06:03.038810 2031 leader_election.cc:288] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe [CANDIDATE]: Term 1 election: Requested vote from peers bcc8fba247f64f4c8d6d4e2aaac951f9 (127.0.97.1:41143), b871109b89de4856a53da7e7cc49af2b (127.0.97.2:37359)
I20260926 03:06:03.039402 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaca44d2cdc249289ff451faf67216d4" candidate_uuid: "f5cd0c81921d44939e7fb31c9a125afe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9"
I20260926 03:06:03.039412 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaca44d2cdc249289ff451faf67216d4" candidate_uuid: "f5cd0c81921d44939e7fb31c9a125afe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b871109b89de4856a53da7e7cc49af2b"
I20260926 03:06:03.039598 1670 raft_consensus.cc:3037] T eaca44d2cdc249289ff451faf67216d4 P bcc8fba247f64f4c8d6d4e2aaac951f9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:03.039626 1808 raft_consensus.cc:3037] T eaca44d2cdc249289ff451faf67216d4 P b871109b89de4856a53da7e7cc49af2b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:03.041661 1670 raft_consensus.cc:2454] T eaca44d2cdc249289ff451faf67216d4 P bcc8fba247f64f4c8d6d4e2aaac951f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5cd0c81921d44939e7fb31c9a125afe in term 1.
I20260926 03:06:03.042120 1879 leader_election.cc:302] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcc8fba247f64f4c8d6d4e2aaac951f9, f5cd0c81921d44939e7fb31c9a125afe; no voters:
I20260926 03:06:03.042393 2031 raft_consensus.cc:2781] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:03.042937 1808 raft_consensus.cc:2454] T eaca44d2cdc249289ff451faf67216d4 P b871109b89de4856a53da7e7cc49af2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5cd0c81921d44939e7fb31c9a125afe in term 1.
I20260926 03:06:03.043059 2031 raft_consensus.cc:686] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe [term 1 LEADER]: Becoming Leader. State: Replica: f5cd0c81921d44939e7fb31c9a125afe, State: Running, Role: LEADER
I20260926 03:06:03.043380 2031 consensus_queue.cc:237] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } }
I20260926 03:06:03.046530 1513 catalog_manager.cc:5495] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe reported cstate change: term changed from 0 to 1, leader changed from <none> to f5cd0c81921d44939e7fb31c9a125afe (127.0.97.3). New cstate: current_term: 1 leader_uuid: "f5cd0c81921d44939e7fb31c9a125afe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } health_report { overall_health: HEALTHY } } }
Sep 26 03:06:03 dist-test-slave-hmbw krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:03.107554 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:03.092976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39621 (local address 127.0.0.1:44366)
0926 03:06:03.093740 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:03.093765 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:03.093946 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:03.094305 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:03.094311 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:03.095603 (+ 1292us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:03.095875 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:03.095887 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:03.097008 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:03.097013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:03.097969 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:03.097981 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:03.098711 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:03.098716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:03.098878 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:03.099587 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:03.099604 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:03.102042 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:03.105057 (+ 3015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:03.105073 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:03.105081 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:03.105485 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:03.105814 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:03.105819 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:03.105822 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:03.106017 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:03.106542 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:03.106552 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:03.106699 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 03:06:03.106840 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 03:06:03.107144 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":4,"thread_start_us":380,"threads_started":1}
I20260926 03:06:03.107630 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:03.092987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44366 (local address 127.0.97.62:39621)
0926 03:06:03.094054 (+ 1067us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:03.094059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:03.094081 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:06:03.094120 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:03.094123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:03.094174 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:03.094262 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:03.096077 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:03.096851 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:03.098113 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:03.098543 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:03.098660 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:03.102257 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:03.102262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:03.102265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:03.102294 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:03.104925 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:03.105598 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:03.105603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:03.105606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:03.105716 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:03.106115 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:03.106120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:03.106123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:03.106393 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:03.106542 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:03.106936 (+ 394us) server_negotiation.cc:299] Negotiation successful
0926 03:06:03.107280 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":143,"server-negotiator.queue_time_us":953,"thread_start_us":814,"threads_started":1}
I20260926 03:06:03.113143 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44366:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 03:06:03.114087 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:06:03.121348 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 8d3ceccbed7f402185f047562809c616 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d013ca7f42164543b7bf182700c2a25d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:06:03.121416 1789 tablet_service.cc:1467] Processing CreateTablet for tablet 8d3ceccbed7f402185f047562809c616 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d013ca7f42164543b7bf182700c2a25d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:06:03.121943 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d3ceccbed7f402185f047562809c616. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:03.121937 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d3ceccbed7f402185f047562809c616. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:03.126461 1929 tablet_service.cc:1467] Processing CreateTablet for tablet 8d3ceccbed7f402185f047562809c616 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d013ca7f42164543b7bf182700c2a25d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:06:03.127136 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d3ceccbed7f402185f047562809c616. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:03.128413 2021 tablet_bootstrap.cc:492] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b: Bootstrap starting.
I20260926 03:06:03.129333 2019 tablet_bootstrap.cc:492] T 8d3ceccbed7f402185f047562809c616 P bcc8fba247f64f4c8d6d4e2aaac951f9: Bootstrap starting.
I20260926 03:06:03.130070 2021 tablet_bootstrap.cc:654] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:03.130926 2019 tablet_bootstrap.cc:654] T 8d3ceccbed7f402185f047562809c616 P bcc8fba247f64f4c8d6d4e2aaac951f9: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:03.139353 2018 tablet_bootstrap.cc:492] T 8d3ceccbed7f402185f047562809c616 P f5cd0c81921d44939e7fb31c9a125afe: Bootstrap starting.
I20260926 03:06:03.140522 2021 tablet_bootstrap.cc:492] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b: No bootstrap required, opened a new log
I20260926 03:06:03.140735 2021 ts_tablet_manager.cc:1397] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.011s
I20260926 03:06:03.140923 2019 tablet_bootstrap.cc:492] T 8d3ceccbed7f402185f047562809c616 P bcc8fba247f64f4c8d6d4e2aaac951f9: No bootstrap required, opened a new log
I20260926 03:06:03.141062 2018 tablet_bootstrap.cc:654] T 8d3ceccbed7f402185f047562809c616 P f5cd0c81921d44939e7fb31c9a125afe: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:03.141170 2019 ts_tablet_manager.cc:1397] T 8d3ceccbed7f402185f047562809c616 P bcc8fba247f64f4c8d6d4e2aaac951f9: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260926 03:06:03.141518 2021 raft_consensus.cc:348] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } }
I20260926 03:06:03.141830 2021 raft_consensus.cc:374] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:03.141966 2021 raft_consensus.cc:729] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b871109b89de4856a53da7e7cc49af2b, State: Initialized, Role: FOLLOWER
I20260926 03:06:03.141896 2019 raft_consensus.cc:348] T 8d3ceccbed7f402185f047562809c616 P bcc8fba247f64f4c8d6d4e2aaac951f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } }
I20260926 03:06:03.142191 2019 raft_consensus.cc:374] T 8d3ceccbed7f402185f047562809c616 P bcc8fba247f64f4c8d6d4e2aaac951f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:03.142293 2019 raft_consensus.cc:729] T 8d3ceccbed7f402185f047562809c616 P bcc8fba247f64f4c8d6d4e2aaac951f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcc8fba247f64f4c8d6d4e2aaac951f9, State: Initialized, Role: FOLLOWER
I20260926 03:06:03.142292 2021 consensus_queue.cc:260] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } }
I20260926 03:06:03.142721 2026 tablet_replica.cc:439] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } } }
I20260926 03:06:03.143051 2026 tablet_replica.cc:442] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:06:03.143568 2021 ts_tablet_manager.cc:1428] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 03:06:03.145030 2019 consensus_queue.cc:260] T 8d3ceccbed7f402185f047562809c616 P bcc8fba247f64f4c8d6d4e2aaac951f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } }
I20260926 03:06:03.147603 2025 tablet_replica.cc:439] T 8d3ceccbed7f402185f047562809c616 P bcc8fba247f64f4c8d6d4e2aaac951f9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } } }
I20260926 03:06:03.147996 2025 tablet_replica.cc:442] T 8d3ceccbed7f402185f047562809c616 P bcc8fba247f64f4c8d6d4e2aaac951f9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:06:03.148676 2019 ts_tablet_manager.cc:1428] T 8d3ceccbed7f402185f047562809c616 P bcc8fba247f64f4c8d6d4e2aaac951f9: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20260926 03:06:03.152817 2018 tablet_bootstrap.cc:492] T 8d3ceccbed7f402185f047562809c616 P f5cd0c81921d44939e7fb31c9a125afe: No bootstrap required, opened a new log
I20260926 03:06:03.153008 2018 ts_tablet_manager.cc:1397] T 8d3ceccbed7f402185f047562809c616 P f5cd0c81921d44939e7fb31c9a125afe: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20260926 03:06:03.153699 2018 raft_consensus.cc:348] T 8d3ceccbed7f402185f047562809c616 P f5cd0c81921d44939e7fb31c9a125afe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } }
I20260926 03:06:03.153932 2018 raft_consensus.cc:374] T 8d3ceccbed7f402185f047562809c616 P f5cd0c81921d44939e7fb31c9a125afe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:03.154017 2018 raft_consensus.cc:729] T 8d3ceccbed7f402185f047562809c616 P f5cd0c81921d44939e7fb31c9a125afe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5cd0c81921d44939e7fb31c9a125afe, State: Initialized, Role: FOLLOWER
I20260926 03:06:03.154251 2018 consensus_queue.cc:260] T 8d3ceccbed7f402185f047562809c616 P f5cd0c81921d44939e7fb31c9a125afe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } }
I20260926 03:06:03.154870 2018 ts_tablet_manager.cc:1428] T 8d3ceccbed7f402185f047562809c616 P f5cd0c81921d44939e7fb31c9a125afe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:06:03.154706 2031 tablet_replica.cc:439] T 8d3ceccbed7f402185f047562809c616 P f5cd0c81921d44939e7fb31c9a125afe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } } }
I20260926 03:06:03.155105 2031 tablet_replica.cc:442] T 8d3ceccbed7f402185f047562809c616 P f5cd0c81921d44939e7fb31c9a125afe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:06:03.421312 2031 consensus_queue.cc:1035] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:03.429266 2031 consensus_queue.cc:1035] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:03.497155 2026 raft_consensus.cc:482] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:06:03.497395 2026 raft_consensus.cc:504] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } }
I20260926 03:06:03.498032 2026 leader_election.cc:288] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcc8fba247f64f4c8d6d4e2aaac951f9 (127.0.97.1:41143), f5cd0c81921d44939e7fb31c9a125afe (127.0.97.3:44393)
I20260926 03:06:03.498649 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3ceccbed7f402185f047562809c616" candidate_uuid: "b871109b89de4856a53da7e7cc49af2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5cd0c81921d44939e7fb31c9a125afe" is_pre_election: true
I20260926 03:06:03.498785 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3ceccbed7f402185f047562809c616" candidate_uuid: "b871109b89de4856a53da7e7cc49af2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" is_pre_election: true
I20260926 03:06:03.498924 1949 raft_consensus.cc:2454] T 8d3ceccbed7f402185f047562809c616 P f5cd0c81921d44939e7fb31c9a125afe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b871109b89de4856a53da7e7cc49af2b in term 0.
I20260926 03:06:03.498993 1670 raft_consensus.cc:2454] T 8d3ceccbed7f402185f047562809c616 P bcc8fba247f64f4c8d6d4e2aaac951f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b871109b89de4856a53da7e7cc49af2b in term 0.
I20260926 03:06:03.499411 1741 leader_election.cc:302] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b871109b89de4856a53da7e7cc49af2b, bcc8fba247f64f4c8d6d4e2aaac951f9, f5cd0c81921d44939e7fb31c9a125afe; no voters:
I20260926 03:06:03.499733 2026 raft_consensus.cc:2781] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:06:03.499861 2026 raft_consensus.cc:482] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:06:03.499963 2026 raft_consensus.cc:3037] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:03.502991 2026 raft_consensus.cc:504] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } }
I20260926 03:06:03.503729 2026 leader_election.cc:288] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b [CANDIDATE]: Term 1 election: Requested vote from peers bcc8fba247f64f4c8d6d4e2aaac951f9 (127.0.97.1:41143), f5cd0c81921d44939e7fb31c9a125afe (127.0.97.3:44393)
I20260926 03:06:03.504252 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3ceccbed7f402185f047562809c616" candidate_uuid: "b871109b89de4856a53da7e7cc49af2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9"
I20260926 03:06:03.504302 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3ceccbed7f402185f047562809c616" candidate_uuid: "b871109b89de4856a53da7e7cc49af2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5cd0c81921d44939e7fb31c9a125afe"
I20260926 03:06:03.504487 1670 raft_consensus.cc:3037] T 8d3ceccbed7f402185f047562809c616 P bcc8fba247f64f4c8d6d4e2aaac951f9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:03.504550 1949 raft_consensus.cc:3037] T 8d3ceccbed7f402185f047562809c616 P f5cd0c81921d44939e7fb31c9a125afe [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:03.506531 1670 raft_consensus.cc:2454] T 8d3ceccbed7f402185f047562809c616 P bcc8fba247f64f4c8d6d4e2aaac951f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b871109b89de4856a53da7e7cc49af2b in term 1.
I20260926 03:06:03.507007 1740 leader_election.cc:302] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b871109b89de4856a53da7e7cc49af2b, bcc8fba247f64f4c8d6d4e2aaac951f9; no voters:
I20260926 03:06:03.507265 2026 raft_consensus.cc:2781] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:03.507565 1949 raft_consensus.cc:2454] T 8d3ceccbed7f402185f047562809c616 P f5cd0c81921d44939e7fb31c9a125afe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b871109b89de4856a53da7e7cc49af2b in term 1.
I20260926 03:06:03.507704 2026 raft_consensus.cc:686] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b [term 1 LEADER]: Becoming Leader. State: Replica: b871109b89de4856a53da7e7cc49af2b, State: Running, Role: LEADER
I20260926 03:06:03.508339 2026 consensus_queue.cc:237] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } }
I20260926 03:06:03.510449 2065 tablet_replica.cc:439] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b: TxnStatusTablet state changed. Reason: New leader b871109b89de4856a53da7e7cc49af2b. Latest consensus state: current_term: 1 leader_uuid: "b871109b89de4856a53da7e7cc49af2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } } }
I20260926 03:06:03.510851 2065 tablet_replica.cc:442] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:06:03.511986 2067 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 03:06:03.512207 2067 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 03:06:03.512282 1513 catalog_manager.cc:5495] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b reported cstate change: term changed from 0 to 1, leader changed from <none> to b871109b89de4856a53da7e7cc49af2b (127.0.97.2). New cstate: current_term: 1 leader_uuid: "b871109b89de4856a53da7e7cc49af2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } health_report { overall_health: UNKNOWN } } }
I20260926 03:06:03.601763 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:03.595507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37359 (local address 127.0.0.1:58458)
0926 03:06:03.595866 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:03.595887 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:03.596046 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:03.596660 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:03.596665 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:03.596678 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:06:03.596866 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:03.596875 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:03.598009 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:03.598015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:03.599061 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:03.599084 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:03.599833 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:03.599838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:03.600005 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:03.600065 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:06:03.601227 (+ 1162us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:06:03.601251 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 03:06:03.601326 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 03:06:03.601486 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188}
I20260926 03:06:03.601811 2070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:03.595589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58458 (local address 127.0.97.2:37359)
0926 03:06:03.596284 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:03.596289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:03.596310 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:06:03.596385 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:03.596388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:03.596473 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:03.596608 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:06:03.597031 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:03.597877 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:03.599207 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:03.599675 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:03.599793 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:03.600504 (+ 711us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:06:03.601111 (+ 607us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:06:03.601238 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:03.601370 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 03:06:03.601464 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":387,"threads_started":1}
I20260926 03:06:03.606247 1670 raft_consensus.cc:1261] T 8d3ceccbed7f402185f047562809c616 P bcc8fba247f64f4c8d6d4e2aaac951f9 [term 1 FOLLOWER]: Refusing update from remote peer b871109b89de4856a53da7e7cc49af2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:03.606464 1949 raft_consensus.cc:1261] T 8d3ceccbed7f402185f047562809c616 P f5cd0c81921d44939e7fb31c9a125afe [term 1 FOLLOWER]: Refusing update from remote peer b871109b89de4856a53da7e7cc49af2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:03.606968 2026 consensus_queue.cc:1035] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:03.607321 2065 consensus_queue.cc:1035] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:03.610566 2031 tablet_replica.cc:439] T 8d3ceccbed7f402185f047562809c616 P f5cd0c81921d44939e7fb31c9a125afe: TxnStatusTablet state changed. Reason: New leader b871109b89de4856a53da7e7cc49af2b. Latest consensus state: current_term: 1 leader_uuid: "b871109b89de4856a53da7e7cc49af2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } } }
I20260926 03:06:03.610839 2031 tablet_replica.cc:442] T 8d3ceccbed7f402185f047562809c616 P f5cd0c81921d44939e7fb31c9a125afe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:06:03.620222 2025 tablet_replica.cc:439] T 8d3ceccbed7f402185f047562809c616 P bcc8fba247f64f4c8d6d4e2aaac951f9: TxnStatusTablet state changed. Reason: New leader b871109b89de4856a53da7e7cc49af2b. Latest consensus state: current_term: 1 leader_uuid: "b871109b89de4856a53da7e7cc49af2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } } }
I20260926 03:06:03.620486 2025 tablet_replica.cc:442] T 8d3ceccbed7f402185f047562809c616 P bcc8fba247f64f4c8d6d4e2aaac951f9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:06:03.623836 2031 tablet_replica.cc:439] T 8d3ceccbed7f402185f047562809c616 P f5cd0c81921d44939e7fb31c9a125afe: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b871109b89de4856a53da7e7cc49af2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } } }
I20260926 03:06:03.624109 2031 tablet_replica.cc:442] T 8d3ceccbed7f402185f047562809c616 P f5cd0c81921d44939e7fb31c9a125afe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:06:03.625133 2065 tablet_replica.cc:439] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b871109b89de4856a53da7e7cc49af2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } } }
I20260926 03:06:03.625414 2065 tablet_replica.cc:442] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:06:03.625577 2026 tablet_replica.cc:439] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b871109b89de4856a53da7e7cc49af2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } } }
I20260926 03:06:03.625801 2026 tablet_replica.cc:442] T 8d3ceccbed7f402185f047562809c616 P b871109b89de4856a53da7e7cc49af2b: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:06:03.626681 2025 tablet_replica.cc:439] T 8d3ceccbed7f402185f047562809c616 P bcc8fba247f64f4c8d6d4e2aaac951f9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b871109b89de4856a53da7e7cc49af2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc8fba247f64f4c8d6d4e2aaac951f9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41143 } } peers { permanent_uuid: "b871109b89de4856a53da7e7cc49af2b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f5cd0c81921d44939e7fb31c9a125afe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44393 } } }
I20260926 03:06:03.626945 2025 tablet_replica.cc:442] T 8d3ceccbed7f402185f047562809c616 P bcc8fba247f64f4c8d6d4e2aaac951f9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:06:03.645321 2070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:03.639732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46217 (local address 127.0.97.2:37359)
0926 03:06:03.639986 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:03.639994 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:03.640473 (+ 479us) server_negotiation.cc:407] Connection header received
0926 03:06:03.640672 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:03.640677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:03.640740 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:03.640832 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:06:03.641260 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:03.642108 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:03.643503 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:03.644037 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:03.644193 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:03.644635 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:06:03.644739 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:06:03.644842 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:03.644951 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 03:06:03.645026 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 03:06:03.645411 2079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:03.639706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37359 (local address 127.0.97.3:46217)
0926 03:06:03.640358 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:03.640377 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:03.640539 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:03.640851 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:03.640855 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:03.640868 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:06:03.641119 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:03.641127 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:03.642240 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:03.642245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:03.643366 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:03.643380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:03.644245 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:03.644251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:03.644393 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:03.644449 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:06:03.644837 (+ 388us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:06:03.644848 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 03:06:03.644918 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 03:06:03.645053 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":327,"threads_started":1}
I20260926 03:06:03.686889 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:03.679861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39879 (local address 127.0.97.3:44393)
0926 03:06:03.680542 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:03.680550 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:03.681184 (+ 634us) server_negotiation.cc:407] Connection header received
0926 03:06:03.681253 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:03.681258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:03.681340 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:03.681428 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:06:03.681954 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:03.683247 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:03.684927 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:03.685412 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:03.685528 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:03.686158 (+ 630us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:06:03.686288 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:06:03.686366 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:03.686552 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 03:06:03.686622 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":378,"threads_started":1}
I20260926 03:06:03.687050 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:03.679849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44393 (local address 127.0.97.2:39879)
0926 03:06:03.680912 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:03.680930 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:03.681074 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:03.681445 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:03.681449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:03.681461 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:06:03.681746 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:03.681760 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:03.683430 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:03.683435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:03.684782 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:03.684797 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:03.685595 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:03.685602 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:03.685773 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:03.685854 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:06:03.686411 (+ 557us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:06:03.686423 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 03:06:03.686521 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 03:06:03.686665 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"thread_start_us":739,"threads_started":1}
I20260926 03:06:03.719940 1513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:06:03.720157 1513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:06:03.723251 1513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 745f1df3f2994915bcd8fe7b399f03ae: Sending DeleteTablet for 3 replicas of tablet eaca44d2cdc249289ff451faf67216d4
I20260926 03:06:03.724067 1649 tablet_service.cc:1514] Processing DeleteTablet for tablet eaca44d2cdc249289ff451faf67216d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:03 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49548
I20260926 03:06:03.724216 1789 tablet_service.cc:1514] Processing DeleteTablet for tablet eaca44d2cdc249289ff451faf67216d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:03 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58442
I20260926 03:06:03.724537 1929 tablet_service.cc:1514] Processing DeleteTablet for tablet eaca44d2cdc249289ff451faf67216d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:03 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51672
I20260926 03:06:03.725126 2087 tablet_replica.cc:330] T eaca44d2cdc249289ff451faf67216d4 P b871109b89de4856a53da7e7cc49af2b: stopping tablet replica
I20260926 03:06:03.725194 2086 tablet_replica.cc:330] T eaca44d2cdc249289ff451faf67216d4 P bcc8fba247f64f4c8d6d4e2aaac951f9: stopping tablet replica
I20260926 03:06:03.725379 2086 raft_consensus.cc:2229] T eaca44d2cdc249289ff451faf67216d4 P bcc8fba247f64f4c8d6d4e2aaac951f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:06:03.725416 2087 raft_consensus.cc:2229] T eaca44d2cdc249289ff451faf67216d4 P b871109b89de4856a53da7e7cc49af2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:06:03.725647 2086 raft_consensus.cc:2258] T eaca44d2cdc249289ff451faf67216d4 P bcc8fba247f64f4c8d6d4e2aaac951f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:03.725852 2087 raft_consensus.cc:2258] T eaca44d2cdc249289ff451faf67216d4 P b871109b89de4856a53da7e7cc49af2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:03.726820 2086 ts_tablet_manager.cc:1905] T eaca44d2cdc249289ff451faf67216d4 P bcc8fba247f64f4c8d6d4e2aaac951f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:03.727941 2087 ts_tablet_manager.cc:1905] T eaca44d2cdc249289ff451faf67216d4 P b871109b89de4856a53da7e7cc49af2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:03.734141 2086 ts_tablet_manager.cc:1918] T eaca44d2cdc249289ff451faf67216d4 P bcc8fba247f64f4c8d6d4e2aaac951f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:06:03.734300 2086 log.cc:1196] T eaca44d2cdc249289ff451faf67216d4 P bcc8fba247f64f4c8d6d4e2aaac951f9: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-0/wal/wals/eaca44d2cdc249289ff451faf67216d4
I20260926 03:06:03.734732 2086 ts_tablet_manager.cc:1939] T eaca44d2cdc249289ff451faf67216d4 P bcc8fba247f64f4c8d6d4e2aaac951f9: Deleting consensus metadata
I20260926 03:06:03.735910 2087 ts_tablet_manager.cc:1918] T eaca44d2cdc249289ff451faf67216d4 P b871109b89de4856a53da7e7cc49af2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:06:03.735926 1497 catalog_manager.cc:4841] TS bcc8fba247f64f4c8d6d4e2aaac951f9 (127.0.97.1:41143): tablet eaca44d2cdc249289ff451faf67216d4 (table test-table [id=fc3e18a5ee3f46b4a3bb54d4d7fb6027]) successfully deleted
I20260926 03:06:03.736174 2087 log.cc:1196] T eaca44d2cdc249289ff451faf67216d4 P b871109b89de4856a53da7e7cc49af2b: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-1/wal/wals/eaca44d2cdc249289ff451faf67216d4
I20260926 03:06:03.736711 2087 ts_tablet_manager.cc:1939] T eaca44d2cdc249289ff451faf67216d4 P b871109b89de4856a53da7e7cc49af2b: Deleting consensus metadata
I20260926 03:06:03.736886 2088 tablet_replica.cc:330] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe: stopping tablet replica
I20260926 03:06:03.737082 2088 raft_consensus.cc:2229] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:06:03.738032 2088 raft_consensus.cc:2258] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:03.738076 1498 catalog_manager.cc:4841] TS b871109b89de4856a53da7e7cc49af2b (127.0.97.2:37359): tablet eaca44d2cdc249289ff451faf67216d4 (table test-table [id=fc3e18a5ee3f46b4a3bb54d4d7fb6027]) successfully deleted
I20260926 03:06:03.739315 2088 ts_tablet_manager.cc:1905] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:03.746804 2088 ts_tablet_manager.cc:1918] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:06:03.746946 2088 log.cc:1196] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391950058021-388-0/minicluster-data/ts-2/wal/wals/eaca44d2cdc249289ff451faf67216d4
I20260926 03:06:03.747365 2088 ts_tablet_manager.cc:1939] T eaca44d2cdc249289ff451faf67216d4 P f5cd0c81921d44939e7fb31c9a125afe: Deleting consensus metadata
I20260926 03:06:03.749203 1498 catalog_manager.cc:4841] TS f5cd0c81921d44939e7fb31c9a125afe (127.0.97.3:44393): tablet eaca44d2cdc249289ff451faf67216d4 (table test-table [id=fc3e18a5ee3f46b4a3bb54d4d7fb6027]) successfully deleted
Sep 26 03:06:03 dist-test-slave-hmbw krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391962, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:06:03.762219 2090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:03.737254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49550 (local address 127.0.97.1:41143)
0926 03:06:03.744215 (+ 6961us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:03.744221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:03.744615 (+ 394us) server_negotiation.cc:407] Connection header received
0926 03:06:03.744911 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:03.744917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:03.745013 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:03.745109 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:03.747213 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:03.748105 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:03.749941 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:03.750354 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:03.750461 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:03.755864 (+ 5403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:03.755878 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:03.755893 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:03.755939 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:03.759080 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:03.759968 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:03.759979 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:03.759983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:03.760158 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:03.760565 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:03.760570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:03.760573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:03.761043 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:03.761343 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:03.761606 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 03:06:03.761849 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6773,"thread_start_us":6629,"threads_started":1}
W20260926 03:06:03.763067 1715 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49550
I20260926 03:06:03.790912 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:03.766781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44378 (local address 127.0.97.62:39621)
0926 03:06:03.775037 (+ 8256us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:03.775041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:03.775065 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:06:03.775123 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:03.775127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:03.775193 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:03.775322 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:03.777256 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:03.778211 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:03.780116 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:03.780828 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:03.780973 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:03.783993 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:03.784000 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:03.784006 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:03.784050 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:03.787681 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:03.788610 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:03.788617 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:03.788620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:03.788779 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:03.789250 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:03.789255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:03.789259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:03.789642 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:03.789902 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:03.790245 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 03:06:03.790493 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8110,"thread_start_us":7835,"threads_started":1}
W20260926 03:06:03.791604 1513 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44378
I20260926 03:06:03.793453 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 1584
I20260926 03:06:03.812249 2090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:03.794845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49554 (local address 127.0.97.1:41143)
0926 03:06:03.795038 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:03.795051 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:03.795993 (+ 942us) server_negotiation.cc:407] Connection header received
0926 03:06:03.796233 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:03.796239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:03.796336 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:03.796399 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:03.797996 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:03.799076 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:03.801148 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:03.801805 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:03.801944 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:03.805209 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:03.805214 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:03.805218 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:03.805257 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:03.808723 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:03.809713 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:03.809719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:03.809723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:03.809884 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:03.810355 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:03.810361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:03.810364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:03.810825 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:03.811077 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:03.811469 (+ 392us) server_negotiation.cc:299] Negotiation successful
0926 03:06:03.811707 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 03:06:03.812865 1715 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49554
W20260926 03:06:03.813570 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 1584: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:06:03.813799 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 1584
I20260926 03:06:03.829955 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 1724
Sep 26 03:06:03 dist-test-slave-hmbw krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391962, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:06:03.848157 2070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:03.830911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58466 (local address 127.0.97.2:37359)
0926 03:06:03.831129 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:03.831135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:03.831156 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:06:03.831226 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:03.831231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:03.831336 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:03.831449 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:03.832949 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:03.834204 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:03.835666 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:03.836350 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:03.836494 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:03.840793 (+ 4299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:03.840806 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:03.840826 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:03.840890 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:03.844887 (+ 3997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:03.845750 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:03.845762 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:03.845765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:03.845938 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:03.846358 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:03.846364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:03.846367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:03.846875 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:03.847227 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:03.847509 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 03:06:03.847760 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 03:06:03.849071 1854 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58466
W20260926 03:06:03.849860 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 1724: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:06:03.850082 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 1724
I20260926 03:06:03.866359 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 1863
Sep 26 03:06:03 dist-test-slave-hmbw krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391962, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:06:03.881470 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:03.867145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51688 (local address 127.0.97.3:44393)
0926 03:06:03.867365 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:03.867371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:03.867562 (+ 191us) server_negotiation.cc:407] Connection header received
0926 03:06:03.867751 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:03.867757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:03.867831 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:03.867917 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:03.869180 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:03.869865 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:03.871208 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:03.871599 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:03.871705 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:03.875735 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:03.875750 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:03.875768 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:03.875805 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:03.878819 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:03.879512 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:03.879519 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:03.879521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:03.879657 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:03.880009 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:03.880013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:03.880015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:03.880339 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:03.880626 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:03.880921 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 03:06:03.881163 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260926 03:06:03.882212 1994 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51688
W20260926 03:06:03.883025 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 1863: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:06:03.883172 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 1863
I20260926 03:06:03.900195 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 1481
I20260926 03:06:03.901454 1553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:04.016201 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 1481
2026-09-26T03:06:04Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5437 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:06:04 dist-test-slave-hmbw krb5kdc[2099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:06:04 dist-test-slave-hmbw krb5kdc[2099](info): set up 2 sockets
Sep 26 03:06:04 dist-test-slave-hmbw krb5kdc[2099](info): commencing operation
krb5kdc: starting...
W20260926 03:06:05.034675 388 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:06:05 dist-test-slave-hmbw krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:06:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:06:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:06:05.226265 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38389
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:05.553591 2112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:05.553902 2112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:05.554016 2112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:05.561412 2112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:06:05.561554 2112 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 03:06:05.561607 2112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:05.561693 2112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:05.561762 2112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:06:05.561846 2112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:06:05.572217 2112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38389
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:05.573546 2112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:05.574955 2112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:05.586658 2120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:05.586786 2117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:05.586692 2118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:05.588085 2112 server_base.cc:1034] running on GCE node
I20260926 03:06:05.588948 2112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:05.590435 2112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:05.591702 2112 hybrid_clock.cc:648] HybridClock initialized: now 1790391965591643 us; error 32 us; skew 500 ppm
I20260926 03:06:05.592051 2112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:05 dist-test-slave-hmbw krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:05.595155 2112 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:06:05.596529 2112 webserver.cc:460] Webserver started at http://127.0.97.62:36761/ using document root <none> and password file <none>
I20260926 03:06:05.597235 2112 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:05.597326 2112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:05.597605 2112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:05.600770 2112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "9b80281939834119bff32889ec34ea5b"
format_stamp: "Formatted at 2026-09-26 03:06:05 on dist-test-slave-hmbw"
server_key: "d9647408eb70806b17c5b75f5a29dd0a"
server_key_iv: "c35f275fe7a433b68ab71830722eae19"
server_key_version: "encryptionkey@0"
I20260926 03:06:05.601553 2112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "9b80281939834119bff32889ec34ea5b"
format_stamp: "Formatted at 2026-09-26 03:06:05 on dist-test-slave-hmbw"
server_key: "d9647408eb70806b17c5b75f5a29dd0a"
server_key_iv: "c35f275fe7a433b68ab71830722eae19"
server_key_version: "encryptionkey@0"
I20260926 03:06:05.607185 2112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 03:06:05.611099 2127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:05.612663 2112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:06:05.612854 2112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "9b80281939834119bff32889ec34ea5b"
format_stamp: "Formatted at 2026-09-26 03:06:05 on dist-test-slave-hmbw"
server_key: "d9647408eb70806b17c5b75f5a29dd0a"
server_key_iv: "c35f275fe7a433b68ab71830722eae19"
server_key_version: "encryptionkey@0"
I20260926 03:06:05.613065 2112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:05.627404 2112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:05.631950 2112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:05.632254 2112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:05.653124 2112 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38389
I20260926 03:06:05.653193 2188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38389 every 8 connection(s)
I20260926 03:06:05.654582 2112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:06:05.658713 2189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:05.664804 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 2112
I20260926 03:06:05.665007 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:06:05.665601 388 external_mini_cluster.cc:1428] Setting key f34e5e22c15aaa413def9d757003f720
I20260926 03:06:05.669749 2189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b80281939834119bff32889ec34ea5b: Bootstrap starting.
I20260926 03:06:05.673921 2189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b80281939834119bff32889ec34ea5b: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:05.675268 2189 log.cc:824] T 00000000000000000000000000000000 P 9b80281939834119bff32889ec34ea5b: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:05.678270 2189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b80281939834119bff32889ec34ea5b: No bootstrap required, opened a new log
Sep 26 03:06:05 dist-test-slave-hmbw krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:05.688244 2189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b80281939834119bff32889ec34ea5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b80281939834119bff32889ec34ea5b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38389 } }
I20260926 03:06:05.688746 2189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b80281939834119bff32889ec34ea5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:05.688854 2189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b80281939834119bff32889ec34ea5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b80281939834119bff32889ec34ea5b, State: Initialized, Role: FOLLOWER
I20260926 03:06:05.689584 2189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b80281939834119bff32889ec34ea5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b80281939834119bff32889ec34ea5b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38389 } }
I20260926 03:06:05.689886 2189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b80281939834119bff32889ec34ea5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:06:05.690026 2189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b80281939834119bff32889ec34ea5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:06:05.690196 2189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b80281939834119bff32889ec34ea5b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:05.692548 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:05.667895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56952 (local address 127.0.97.62:38389)
0926 03:06:05.668869 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:05.668885 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:05.669306 (+ 421us) server_negotiation.cc:407] Connection header received
0926 03:06:05.671838 (+ 2532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:05.671871 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:05.672451 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:05.672936 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:05.674390 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:05.676317 (+ 1927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:05.677683 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:05.678400 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:05.678598 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:05.683245 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:05.683256 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:05.683286 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:05.683343 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:05.687424 (+ 4081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:05.688267 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:05.688277 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:05.688285 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:05.688496 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:05.688985 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:05.688990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:05.688993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:05.689760 (+ 767us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:05.690059 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:05.690571 (+ 512us) server_negotiation.cc:299] Negotiation successful
0926 03:06:05.691051 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":445,"threads_started":1}
I20260926 03:06:05.693526 2189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b80281939834119bff32889ec34ea5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b80281939834119bff32889ec34ea5b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38389 } }
I20260926 03:06:05.694144 2189 leader_election.cc:302] T 00000000000000000000000000000000 P 9b80281939834119bff32889ec34ea5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b80281939834119bff32889ec34ea5b; no voters:
I20260926 03:06:05.694808 2189 leader_election.cc:288] T 00000000000000000000000000000000 P 9b80281939834119bff32889ec34ea5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:06:05.694973 2194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b80281939834119bff32889ec34ea5b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:05.695701 2194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b80281939834119bff32889ec34ea5b [term 1 LEADER]: Becoming Leader. State: Replica: 9b80281939834119bff32889ec34ea5b, State: Running, Role: LEADER
I20260926 03:06:05.696301 2194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b80281939834119bff32889ec34ea5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b80281939834119bff32889ec34ea5b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38389 } }
I20260926 03:06:05.697721 2189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b80281939834119bff32889ec34ea5b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:06:05.699177 2196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b80281939834119bff32889ec34ea5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b80281939834119bff32889ec34ea5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b80281939834119bff32889ec34ea5b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38389 } } }
I20260926 03:06:05.699589 2196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b80281939834119bff32889ec34ea5b [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:05.702072 2195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b80281939834119bff32889ec34ea5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b80281939834119bff32889ec34ea5b. Latest consensus state: current_term: 1 leader_uuid: "9b80281939834119bff32889ec34ea5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b80281939834119bff32889ec34ea5b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38389 } } }
I20260926 03:06:05.702476 2195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b80281939834119bff32889ec34ea5b [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:05.706738 2201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:06:05.712138 2201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:06:05.721810 2201 catalog_manager.cc:1347] Generated new cluster ID: d0a1864bf5094bb1ac888064c12eecd5
I20260926 03:06:05.721952 2201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:06:05.731113 2201 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:06:05.732487 2201 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:06:05.740988 2201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b80281939834119bff32889ec34ea5b: Generated new TSK 0
I20260926 03:06:05.741905 2201 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:06:05.799036 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38389
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:06.156925 2215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:06.157224 2215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:06.157325 2215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:06.164942 2215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:06.165073 2215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:06.165161 2215 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 03:06:06.165207 2215 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:06:06.165275 2215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:06:06.175989 2215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:06.177356 2215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:06.178660 2215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:06.191738 2220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:06.191704 2223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:06.191674 2221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:06.192611 2215 server_base.cc:1034] running on GCE node
I20260926 03:06:06.193239 2215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:06.194190 2215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:06.195402 2215 hybrid_clock.cc:648] HybridClock initialized: now 1790391966195322 us; error 75 us; skew 500 ppm
I20260926 03:06:06.195737 2215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:06 dist-test-slave-hmbw krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:06.203866 2215 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:06:06.205427 2215 webserver.cc:460] Webserver started at http://127.0.97.1:36081/ using document root <none> and password file <none>
I20260926 03:06:06.206285 2215 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:06.206430 2215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:06.206831 2215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:06.210153 2215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-0/data/instance:
uuid: "b15866fca8ae4ade8630d4f36ef468d4"
format_stamp: "Formatted at 2026-09-26 03:06:06 on dist-test-slave-hmbw"
server_key: "0f4fa21dcec5379de4707b01c8a14d4b"
server_key_iv: "ed1bdfcbf71dc527b3bf895826b46850"
server_key_version: "encryptionkey@0"
I20260926 03:06:06.210989 2215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b15866fca8ae4ade8630d4f36ef468d4"
format_stamp: "Formatted at 2026-09-26 03:06:06 on dist-test-slave-hmbw"
server_key: "0f4fa21dcec5379de4707b01c8a14d4b"
server_key_iv: "ed1bdfcbf71dc527b3bf895826b46850"
server_key_version: "encryptionkey@0"
I20260926 03:06:06.216428 2215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:06:06.220222 2230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:06.221571 2215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260926 03:06:06.221756 2215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "b15866fca8ae4ade8630d4f36ef468d4"
format_stamp: "Formatted at 2026-09-26 03:06:06 on dist-test-slave-hmbw"
server_key: "0f4fa21dcec5379de4707b01c8a14d4b"
server_key_iv: "ed1bdfcbf71dc527b3bf895826b46850"
server_key_version: "encryptionkey@0"
I20260926 03:06:06.221942 2215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:06.241678 2215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:06.246034 2215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:06.246328 2215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:06.249058 2215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:06.249172 2215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:06.249289 2215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:06.249378 2215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:06:06 dist-test-slave-hmbw krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:06.275399 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:06.249991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48895 (local address 127.0.97.62:38389)
0926 03:06:06.250409 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:06.250413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:06.251849 (+ 1436us) server_negotiation.cc:407] Connection header received
0926 03:06:06.253839 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:06.253845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:06.253944 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:06.254048 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:06.256995 (+ 2947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:06.257978 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:06.260459 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:06.260947 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:06.261078 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:06.266275 (+ 5197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:06.266284 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:06.266288 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:06.266337 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:06.269734 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:06.270870 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:06.270876 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:06.270880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:06.271038 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:06.271616 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:06.271622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:06.271625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:06.272062 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:06.272284 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:06.274786 (+ 2502us) server_negotiation.cc:299] Negotiation successful
0926 03:06:06.275029 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":216,"threads_started":1}
I20260926 03:06:06.275594 2240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:06.250745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38389 (local address 127.0.97.1:48895)
0926 03:06:06.251688 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:06.251730 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:06.253531 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:06.254562 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:06.254581 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:06.255619 (+ 1038us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:06.256722 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:06.256761 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:06.258222 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:06.258232 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:06.260289 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:06.260311 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:06.261263 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:06.261272 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:06.261630 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:06.262568 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:06.262615 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:06.265883 (+ 3268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:06.269899 (+ 4016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:06.269912 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:06.269927 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:06.270647 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:06.271179 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:06.271182 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:06.271185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:06.271451 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:06.272317 (+ 866us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:06.272330 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:06.272927 (+ 597us) client_negotiation.cc:769] Sending connection context
0926 03:06:06.273306 (+ 379us) client_negotiation.cc:240] Negotiation successful
0926 03:06:06.273931 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":453,"threads_started":1}
I20260926 03:06:06.290673 2215 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45529
I20260926 03:06:06.290721 2350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45529 every 8 connection(s)
I20260926 03:06:06.292076 2215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:06:06.292670 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 2215
I20260926 03:06:06.292860 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:06:06.293676 388 external_mini_cluster.cc:1428] Setting key 25658837e4ef1db7ce5a512be28b6761
I20260926 03:06:06.294447 2351 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38389
I20260926 03:06:06.294720 2351 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:06.295650 2351 heartbeater.cc:511] Master 127.0.97.62:38389 requested a full tablet report, sending...
I20260926 03:06:06.298074 2144 ts_manager.cc:194] Registered new tserver with Master: b15866fca8ae4ade8630d4f36ef468d4 (127.0.97.1:45529)
I20260926 03:06:06.298646 2339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:06.284832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38389 (local address 127.0.97.1:34059)
0926 03:06:06.285409 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:06.285432 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:06.285633 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:06.285981 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:06.285986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:06.286435 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:06.286912 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:06.286927 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:06.288198 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:06.288202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:06.289394 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:06.289420 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:06.290312 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:06.290319 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:06.290493 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:06.291194 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:06.291211 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:06.291967 (+ 756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:06.295688 (+ 3721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:06.295693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:06.295697 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:06.296219 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:06.296682 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:06.296687 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:06.296691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:06.296839 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:06.297484 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:06.297489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:06.297695 (+ 206us) client_negotiation.cc:769] Sending connection context
0926 03:06:06.297830 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 03:06:06.298170 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":551,"threads_started":1}
I20260926 03:06:06.299732 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:06.285192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34059 (local address 127.0.97.62:38389)
0926 03:06:06.285377 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:06.285383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:06.285518 (+ 135us) server_negotiation.cc:407] Connection header received
0926 03:06:06.285783 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:06.285789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:06.285859 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:06.285939 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:06.287068 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:06.288003 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:06.289563 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:06.290112 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:06.290229 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:06.292168 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:06.292173 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:06.292177 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:06.292218 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:06.295504 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:06.296391 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:06.296395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:06.296399 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:06.296549 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:06.296984 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:06.296989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:06.296992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:06.297348 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:06.298915 (+ 1567us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:06.299072 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 03:06:06.299348 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":68}
I20260926 03:06:06.300488 2144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48895
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:06:06.340281 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38389
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:06.669425 2355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:06.669888 2355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:06.670060 2355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:06.677835 2355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:06.677966 2355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:06.678068 2355 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 03:06:06.678126 2355 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:06:06.678197 2355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:06:06.689206 2355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:06.690990 2355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:06.692538 2355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:06.707448 2360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:06.707463 2361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:06.707511 2363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:06.708355 2355 server_base.cc:1034] running on GCE node
I20260926 03:06:06.709002 2355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:06.710069 2355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:06.711375 2355 hybrid_clock.cc:648] HybridClock initialized: now 1790391966711206 us; error 85 us; skew 500 ppm
I20260926 03:06:06.711728 2355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:06 dist-test-slave-hmbw krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:06.715281 2355 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:06:06.717511 2355 webserver.cc:460] Webserver started at http://127.0.97.2:37345/ using document root <none> and password file <none>
I20260926 03:06:06.718281 2355 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:06.718400 2355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:06.718794 2355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:06.722144 2355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-1/data/instance:
uuid: "ffc2110a66234d14a58b23e4ff09ea0a"
format_stamp: "Formatted at 2026-09-26 03:06:06 on dist-test-slave-hmbw"
server_key: "583147921d375435a1e745184b3ceae5"
server_key_iv: "652d6c0b822a358af6e7c2b0bc56db65"
server_key_version: "encryptionkey@0"
I20260926 03:06:06.723098 2355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ffc2110a66234d14a58b23e4ff09ea0a"
format_stamp: "Formatted at 2026-09-26 03:06:06 on dist-test-slave-hmbw"
server_key: "583147921d375435a1e745184b3ceae5"
server_key_iv: "652d6c0b822a358af6e7c2b0bc56db65"
server_key_version: "encryptionkey@0"
I20260926 03:06:06.729660 2355 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 03:06:06.734988 2370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:06.737375 2355 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:06:06.737731 2355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "ffc2110a66234d14a58b23e4ff09ea0a"
format_stamp: "Formatted at 2026-09-26 03:06:06 on dist-test-slave-hmbw"
server_key: "583147921d375435a1e745184b3ceae5"
server_key_iv: "652d6c0b822a358af6e7c2b0bc56db65"
server_key_version: "encryptionkey@0"
I20260926 03:06:06.738049 2355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:06.759567 2355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:06.765148 2355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:06.765450 2355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:06.769315 2355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:06.769610 2355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:06.769784 2355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:06.769871 2355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:06:06 dist-test-slave-hmbw krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:06.796002 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:06.769590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48471 (local address 127.0.97.62:38389)
0926 03:06:06.769901 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:06.769908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:06.771383 (+ 1475us) server_negotiation.cc:407] Connection header received
0926 03:06:06.773715 (+ 2332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:06.773722 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:06.773832 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:06.773963 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:06.776682 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:06.777868 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:06.780674 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:06.781280 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:06.781439 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:06.786752 (+ 5313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:06.786758 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:06.786762 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:06.786801 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:06.790557 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:06.791792 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:06.791799 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:06.791802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:06.791977 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:06.792527 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:06.792533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:06.792536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:06.793537 (+ 1001us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:06.793831 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:06.795458 (+ 1627us) server_negotiation.cc:299] Negotiation successful
0926 03:06:06.795683 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260926 03:06:06.798676 2379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:06.769987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38389 (local address 127.0.97.2:48471)
0926 03:06:06.771062 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:06.771124 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:06.773295 (+ 2171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:06.774385 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:06.774397 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:06.775349 (+ 952us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:06.776392 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:06.776429 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:06.778103 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:06.778109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:06.780449 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:06.780472 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:06.781660 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:06.781671 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:06.782071 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:06.783095 (+ 1024us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:06.783134 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:06.786441 (+ 3307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:06.790760 (+ 4319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:06.790772 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:06.790796 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:06.791530 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:06.792120 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:06.792127 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:06.792130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:06.792390 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:06.793880 (+ 1490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:06.793962 (+ 82us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:06.794796 (+ 834us) client_negotiation.cc:769] Sending connection context
0926 03:06:06.795265 (+ 469us) client_negotiation.cc:240] Negotiation successful
0926 03:06:06.795993 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":486,"threads_started":1}
I20260926 03:06:06.817112 2355 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39487
I20260926 03:06:06.817122 2489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39487 every 8 connection(s)
I20260926 03:06:06.818689 2355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:06:06.819526 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 2355
I20260926 03:06:06.819686 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:06:06.820248 388 external_mini_cluster.cc:1428] Setting key 721b6db8371d7e1f8bcd6f326116c0cf
I20260926 03:06:06.822238 2490 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38389
I20260926 03:06:06.822587 2490 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:06.823575 2490 heartbeater.cc:511] Master 127.0.97.62:38389 requested a full tablet report, sending...
I20260926 03:06:06.825407 2144 ts_manager.cc:194] Registered new tserver with Master: ffc2110a66234d14a58b23e4ff09ea0a (127.0.97.2:39487)
I20260926 03:06:06.826637 2144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48471
I20260926 03:06:06.827618 2477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:06.807127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38389 (local address 127.0.97.2:58871)
0926 03:06:06.811192 (+ 4065us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:06.811223 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:06.811395 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:06.811815 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:06.811822 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:06.812495 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:06.812929 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:06.812945 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:06.814753 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:06.814760 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:06.815897 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:06.815913 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:06.817413 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:06.817424 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:06.817939 (+ 515us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:06.818680 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:06.818700 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:06.819974 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:06.824221 (+ 4247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:06.824226 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:06.824231 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:06.824787 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:06.825226 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:06.825231 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:06.825234 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:06.825390 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:06.826364 (+ 974us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:06.826368 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:06.826588 (+ 220us) client_negotiation.cc:769] Sending connection context
0926 03:06:06.826728 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 03:06:06.827039 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3894,"thread_start_us":3615,"threads_started":1}
I20260926 03:06:06.828524 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:06.807125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58871 (local address 127.0.97.62:38389)
0926 03:06:06.807322 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:06.807328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:06.811347 (+ 4019us) server_negotiation.cc:407] Connection header received
0926 03:06:06.811523 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:06.811529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:06.811619 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:06.811733 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:06.813259 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:06.814573 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:06.816116 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:06.817057 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:06.817199 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:06.820373 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:06.820384 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:06.820388 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:06.820435 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:06.824035 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:06.824929 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:06.824934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:06.824938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:06.825102 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:06.825566 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:06.825571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:06.825574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:06.826138 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:06.826300 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:06.827987 (+ 1687us) server_negotiation.cc:299] Negotiation successful
0926 03:06:06.828208 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:06:06.867609 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38389
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:07.227806 2494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:07.228103 2494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:07.228212 2494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:07.236335 2494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:07.236460 2494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:07.236546 2494 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 03:06:07.236595 2494 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:06:07.236662 2494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:06:07.246994 2494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:07.248549 2494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:07.249981 2494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:07.265309 2499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:07.265337 2502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:07.265436 2494 server_base.cc:1034] running on GCE node
W20260926 03:06:07.265301 2500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:07.266521 2494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:07.267601 2494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:07.268848 2494 hybrid_clock.cc:648] HybridClock initialized: now 1790391967268803 us; error 35 us; skew 500 ppm
I20260926 03:06:07.269183 2494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:07 dist-test-slave-hmbw krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:07.272639 2494 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:06:07.274263 2494 webserver.cc:460] Webserver started at http://127.0.97.3:38005/ using document root <none> and password file <none>
I20260926 03:06:07.275027 2494 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:07.275163 2494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:07.275508 2494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:07.278896 2494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-2/data/instance:
uuid: "278332e68f62492e90cbeb5802b32f8e"
format_stamp: "Formatted at 2026-09-26 03:06:07 on dist-test-slave-hmbw"
server_key: "52993c64f433720d03af88a6a7cf0d09"
server_key_iv: "60be7073a8c52d236f66d0385ba166b5"
server_key_version: "encryptionkey@0"
I20260926 03:06:07.279743 2494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-2/wal/instance:
uuid: "278332e68f62492e90cbeb5802b32f8e"
format_stamp: "Formatted at 2026-09-26 03:06:07 on dist-test-slave-hmbw"
server_key: "52993c64f433720d03af88a6a7cf0d09"
server_key_iv: "60be7073a8c52d236f66d0385ba166b5"
server_key_version: "encryptionkey@0"
I20260926 03:06:07.285267 2494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 03:06:07.289043 2509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:07.290866 2494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 03:06:07.291050 2494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "278332e68f62492e90cbeb5802b32f8e"
format_stamp: "Formatted at 2026-09-26 03:06:07 on dist-test-slave-hmbw"
server_key: "52993c64f433720d03af88a6a7cf0d09"
server_key_iv: "60be7073a8c52d236f66d0385ba166b5"
server_key_version: "encryptionkey@0"
I20260926 03:06:07.291291 2494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:07.305435 2351 heartbeater.cc:503] Master 127.0.97.62:38389 was elected leader, sending a full tablet report...
I20260926 03:06:07.318487 2494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:07.324191 2494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:07.324528 2494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:07.327678 2494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:07.327816 2494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:07.327951 2494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:07.328025 2494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:06:07 dist-test-slave-hmbw krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:07.352490 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:07.328137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55217 (local address 127.0.97.62:38389)
0926 03:06:07.328316 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:07.328323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:07.329940 (+ 1617us) server_negotiation.cc:407] Connection header received
0926 03:06:07.332136 (+ 2196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:07.332142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:07.332233 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:07.332317 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:07.334469 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.335484 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.338531 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.339128 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.339269 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:07.344167 (+ 4898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:07.344177 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:07.344183 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:07.344237 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:07.348063 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:07.349091 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:07.349099 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:07.349103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:07.349306 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:07.350048 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:07.350054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:07.350058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:07.350550 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:07.350711 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:07.351873 (+ 1162us) server_negotiation.cc:299] Negotiation successful
0926 03:06:07.352133 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 03:06:07.354157 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:07.328860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38389 (local address 127.0.97.3:55217)
0926 03:06:07.329748 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:07.329796 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:07.331839 (+ 2043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:07.332583 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:07.332592 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:07.333317 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:07.334245 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:07.334285 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.335658 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.335665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:07.338299 (+ 2634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:07.338322 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.339438 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.339450 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:07.339812 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:07.340740 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:07.340781 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:07.343855 (+ 3074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:07.348277 (+ 4422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:07.348291 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:07.348313 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:07.348926 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:07.349506 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:07.349511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:07.349514 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:07.349893 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:07.350750 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:07.350761 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:07.351330 (+ 569us) client_negotiation.cc:769] Sending connection context
0926 03:06:07.351722 (+ 392us) client_negotiation.cc:240] Negotiation successful
0926 03:06:07.352385 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":400,"threads_started":1}
I20260926 03:06:07.376082 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:07.361031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38389 (local address 127.0.97.3:44079)
0926 03:06:07.361755 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:07.361771 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:07.361938 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:07.362437 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:07.362440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:07.362923 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:07.363317 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:07.363333 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.364565 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.364572 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:07.365767 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:07.365780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.366661 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.366667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:07.366874 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:07.367754 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:07.367779 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:07.369031 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:07.372394 (+ 3363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:07.372404 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:07.372408 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:07.372913 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:07.373368 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:07.373374 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:07.373379 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:07.373545 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:07.374352 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:07.374357 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:07.374608 (+ 251us) client_negotiation.cc:769] Sending connection context
0926 03:06:07.374806 (+ 198us) client_negotiation.cc:240] Negotiation successful
0926 03:06:07.375262 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":458,"threads_started":1}
I20260926 03:06:07.376220 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:07.360762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44079 (local address 127.0.97.62:38389)
0926 03:06:07.360944 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:07.360950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:07.361868 (+ 918us) server_negotiation.cc:407] Connection header received
0926 03:06:07.362055 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:07.362061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:07.362167 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:07.362275 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:07.363514 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.364408 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.365920 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.366424 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.366579 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:07.369313 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:07.369319 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:07.369322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:07.369356 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:07.372242 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:07.373036 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:07.373041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:07.373045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:07.373220 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:07.373692 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:07.373697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:07.373700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:07.374092 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:07.374269 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:07.375510 (+ 1241us) server_negotiation.cc:299] Negotiation successful
0926 03:06:07.375854 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 03:06:07.379948 2494 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34285
I20260926 03:06:07.380030 2628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34285 every 8 connection(s)
I20260926 03:06:07.381400 2494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:06:07.383790 2629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38389
I20260926 03:06:07.384030 2629 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:07.384696 2629 heartbeater.cc:511] Master 127.0.97.62:38389 requested a full tablet report, sending...
I20260926 03:06:07.386174 2144 ts_manager.cc:194] Registered new tserver with Master: 278332e68f62492e90cbeb5802b32f8e (127.0.97.3:34285)
I20260926 03:06:07.387049 2144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55217
I20260926 03:06:07.387871 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 2494
I20260926 03:06:07.388012 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:06:07.388518 388 external_mini_cluster.cc:1428] Setting key 78b3164ede1958272985a28c8de52723
I20260926 03:06:07.391747 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 03:06:07 dist-test-slave-hmbw krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 03:06:07 dist-test-slave-hmbw krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:07.438421 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:07.424157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56966 (local address 127.0.97.62:38389)
0926 03:06:07.424414 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:07.424421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:07.424606 (+ 185us) server_negotiation.cc:407] Connection header received
0926 03:06:07.424817 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:07.424822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:07.424894 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:07.425000 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:07.426631 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.427382 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.428653 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.429073 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.429170 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:07.432979 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:07.432984 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:07.432988 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:07.433031 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:07.435714 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:07.436462 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:07.436466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:07.436469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:07.436608 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:07.437028 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:07.437033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:07.437035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:07.437330 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:07.437515 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:07.437831 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 03:06:07.438017 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260926 03:06:07.442744 2144 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:06:07.449653 2144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:06:07.471781 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:07.463107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39730 (local address 127.0.97.2:39487)
0926 03:06:07.463654 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:07.463661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:07.463861 (+ 200us) server_negotiation.cc:407] Connection header received
0926 03:06:07.464163 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:07.464171 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:07.464531 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:07.464761 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:07.465633 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.467033 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.469365 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.470561 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.470694 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:07.470902 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:07.471313 (+ 411us) server_negotiation.cc:299] Negotiation successful
0926 03:06:07.471485 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":298,"threads_started":1}
I20260926 03:06:07.472043 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:07.462965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39487 (local address 127.0.0.1:39730)
0926 03:06:07.463678 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:07.463731 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:07.463918 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:07.465011 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:07.465019 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:07.465050 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:07.465413 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:07.465425 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.467189 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.467196 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:07.469155 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:07.469169 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.470781 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.470788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:07.470999 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:07.471096 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 03:06:07.471246 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 03:06:07.471452 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":23,"thread_start_us":332,"threads_started":1}
I20260926 03:06:07.472769 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:07.464141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45529 (local address 127.0.0.1:60564)
0926 03:06:07.464652 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:07.464741 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:07.464866 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:07.465958 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:07.465963 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:07.465988 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:07.466381 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:07.466392 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.468274 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.468280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:07.470310 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:07.470324 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.472001 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.472007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:07.472171 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:07.472202 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 03:06:07.472268 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 03:06:07.472404 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":253,"threads_started":1}
I20260926 03:06:07.473742 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:07.464486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60564 (local address 127.0.97.1:45529)
0926 03:06:07.465332 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:07.465341 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:07.465369 (+ 28us) server_negotiation.cc:407] Connection header received
0926 03:06:07.465458 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:07.465479 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:07.465750 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:07.466595 (+ 845us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:07.466795 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.468137 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.470478 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.471762 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.473001 (+ 1239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:07.473210 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:07.473281 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 03:06:07.473456 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":557,"threads_started":1}
I20260926 03:06:07.475255 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:07.465272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34285 (local address 127.0.0.1:34764)
0926 03:06:07.465779 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:07.465801 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:07.465899 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:07.468214 (+ 2315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:07.468219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:07.468246 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:07.468668 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:07.468678 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.470552 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.470557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:07.472687 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:07.472701 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.474423 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.474429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:07.474633 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:07.474666 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:06:07.474788 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 03:06:07.474924 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":282,"threads_started":1}
I20260926 03:06:07.475620 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:07.465017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34764 (local address 127.0.97.3:34285)
0926 03:06:07.467641 (+ 2624us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:07.467649 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:07.467681 (+ 32us) server_negotiation.cc:407] Connection header received
0926 03:06:07.467768 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:07.467776 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:07.468007 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:07.468205 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:07.468895 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.470428 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.472889 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.474213 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.474317 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:07.474530 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:07.474938 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 03:06:07.475104 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2458,"thread_start_us":2294,"threads_started":1}
I20260926 03:06:07.476112 2420 tablet_service.cc:1467] Processing CreateTablet for tablet baaf8dd1595e491b93c4c010d132ccff (DEFAULT_TABLE table=test-table [id=29d311c7122e457fb1309720bcd47356]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:07.477531 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baaf8dd1595e491b93c4c010d132ccff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:07.477975 2281 tablet_service.cc:1467] Processing CreateTablet for tablet baaf8dd1595e491b93c4c010d132ccff (DEFAULT_TABLE table=test-table [id=29d311c7122e457fb1309720bcd47356]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:07.479403 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baaf8dd1595e491b93c4c010d132ccff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:07.479244 2559 tablet_service.cc:1467] Processing CreateTablet for tablet baaf8dd1595e491b93c4c010d132ccff (DEFAULT_TABLE table=test-table [id=29d311c7122e457fb1309720bcd47356]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:07.480690 2559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baaf8dd1595e491b93c4c010d132ccff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:07.491060 2648 tablet_bootstrap.cc:492] T baaf8dd1595e491b93c4c010d132ccff P ffc2110a66234d14a58b23e4ff09ea0a: Bootstrap starting.
I20260926 03:06:07.494897 2650 tablet_bootstrap.cc:492] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e: Bootstrap starting.
I20260926 03:06:07.495568 2648 tablet_bootstrap.cc:654] T baaf8dd1595e491b93c4c010d132ccff P ffc2110a66234d14a58b23e4ff09ea0a: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:07.496832 2649 tablet_bootstrap.cc:492] T baaf8dd1595e491b93c4c010d132ccff P b15866fca8ae4ade8630d4f36ef468d4: Bootstrap starting.
I20260926 03:06:07.497133 2648 log.cc:824] T baaf8dd1595e491b93c4c010d132ccff P ffc2110a66234d14a58b23e4ff09ea0a: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:07.498674 2650 tablet_bootstrap.cc:654] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:07.499980 2650 log.cc:824] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:07.500595 2649 tablet_bootstrap.cc:654] T baaf8dd1595e491b93c4c010d132ccff P b15866fca8ae4ade8630d4f36ef468d4: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:07.500856 2648 tablet_bootstrap.cc:492] T baaf8dd1595e491b93c4c010d132ccff P ffc2110a66234d14a58b23e4ff09ea0a: No bootstrap required, opened a new log
I20260926 03:06:07.501195 2648 ts_tablet_manager.cc:1397] T baaf8dd1595e491b93c4c010d132ccff P ffc2110a66234d14a58b23e4ff09ea0a: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260926 03:06:07.502018 2649 log.cc:824] T baaf8dd1595e491b93c4c010d132ccff P b15866fca8ae4ade8630d4f36ef468d4: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:07.502957 2650 tablet_bootstrap.cc:492] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e: No bootstrap required, opened a new log
I20260926 03:06:07.503216 2650 ts_tablet_manager.cc:1397] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260926 03:06:07.510605 2649 tablet_bootstrap.cc:492] T baaf8dd1595e491b93c4c010d132ccff P b15866fca8ae4ade8630d4f36ef468d4: No bootstrap required, opened a new log
I20260926 03:06:07.510949 2649 ts_tablet_manager.cc:1397] T baaf8dd1595e491b93c4c010d132ccff P b15866fca8ae4ade8630d4f36ef468d4: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.004s
I20260926 03:06:07.511050 2650 raft_consensus.cc:348] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:07.511441 2650 raft_consensus.cc:374] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:07.511529 2650 raft_consensus.cc:729] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 278332e68f62492e90cbeb5802b32f8e, State: Initialized, Role: FOLLOWER
I20260926 03:06:07.511821 2648 raft_consensus.cc:348] T baaf8dd1595e491b93c4c010d132ccff P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:07.512101 2650 consensus_queue.cc:260] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:07.512248 2648 raft_consensus.cc:374] T baaf8dd1595e491b93c4c010d132ccff P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:07.512343 2648 raft_consensus.cc:729] T baaf8dd1595e491b93c4c010d132ccff P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffc2110a66234d14a58b23e4ff09ea0a, State: Initialized, Role: FOLLOWER
I20260926 03:06:07.513155 2629 heartbeater.cc:503] Master 127.0.97.62:38389 was elected leader, sending a full tablet report...
I20260926 03:06:07.513012 2648 consensus_queue.cc:260] T baaf8dd1595e491b93c4c010d132ccff P ffc2110a66234d14a58b23e4ff09ea0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:07.513864 2650 ts_tablet_manager.cc:1428] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
I20260926 03:06:07.515822 2648 ts_tablet_manager.cc:1428] T baaf8dd1595e491b93c4c010d132ccff P ffc2110a66234d14a58b23e4ff09ea0a: Time spent starting tablet: real 0.015s user 0.009s sys 0.006s
I20260926 03:06:07.517410 2490 heartbeater.cc:503] Master 127.0.97.62:38389 was elected leader, sending a full tablet report...
I20260926 03:06:07.520784 2649 raft_consensus.cc:348] T baaf8dd1595e491b93c4c010d132ccff P b15866fca8ae4ade8630d4f36ef468d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:07.521194 2649 raft_consensus.cc:374] T baaf8dd1595e491b93c4c010d132ccff P b15866fca8ae4ade8630d4f36ef468d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:07.521292 2649 raft_consensus.cc:729] T baaf8dd1595e491b93c4c010d132ccff P b15866fca8ae4ade8630d4f36ef468d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b15866fca8ae4ade8630d4f36ef468d4, State: Initialized, Role: FOLLOWER
I20260926 03:06:07.521864 2649 consensus_queue.cc:260] T baaf8dd1595e491b93c4c010d132ccff P b15866fca8ae4ade8630d4f36ef468d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:07.523564 2649 ts_tablet_manager.cc:1428] T baaf8dd1595e491b93c4c010d132ccff P b15866fca8ae4ade8630d4f36ef468d4: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260926 03:06:07.550800 2352 tablet.cc:2369] T baaf8dd1595e491b93c4c010d132ccff P b15866fca8ae4ade8630d4f36ef468d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:06:07.577082 2491 tablet.cc:2369] T baaf8dd1595e491b93c4c010d132ccff P ffc2110a66234d14a58b23e4ff09ea0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:06:07.632961 2630 tablet.cc:2369] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:06:07.963591 2654 raft_consensus.cc:482] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:06:07.963860 2654 raft_consensus.cc:504] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:07.965528 2654 leader_election.cc:288] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffc2110a66234d14a58b23e4ff09ea0a (127.0.97.2:39487), b15866fca8ae4ade8630d4f36ef468d4 (127.0.97.1:45529)
I20260926 03:06:07.971477 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:07.965941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43821 (local address 127.0.97.2:39487)
0926 03:06:07.966114 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:07.966119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:07.966456 (+ 337us) server_negotiation.cc:407] Connection header received
0926 03:06:07.966636 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:07.966642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:07.966736 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:07.966856 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:07.967543 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.968247 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.969818 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.970624 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.970742 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:07.970844 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:07.971105 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 03:06:07.971225 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:06:07.971611 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:07.965766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39487 (local address 127.0.97.3:43821)
0926 03:06:07.966279 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:07.966299 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:07.966463 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:07.966883 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:07.966889 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:07.966921 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:07.967380 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:07.967391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.968389 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.968393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:07.969665 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:07.969676 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.970786 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.970792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:07.970959 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:07.970984 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 03:06:07.971066 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 03:06:07.971204 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":3,"thread_start_us":290,"threads_started":1}
I20260926 03:06:07.972142 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:07.966527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45529 (local address 127.0.97.3:53323)
0926 03:06:07.967351 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:07.967369 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:07.967492 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:07.967796 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:07.967801 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:07.967824 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:07.968069 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:07.968077 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.969015 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.969019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:07.970327 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:07.970338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.971455 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.971461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:07.971651 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:07.971688 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 03:06:07.971777 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 03:06:07.971898 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":23,"thread_start_us":508,"threads_started":1}
I20260926 03:06:07.972153 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:07.966904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53323 (local address 127.0.97.1:45529)
0926 03:06:07.967112 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:07.967118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:07.967459 (+ 341us) server_negotiation.cc:407] Connection header received
0926 03:06:07.967612 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:07.967617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:07.967684 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:07.967768 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:07.968214 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.968880 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.970474 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:07.971236 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:07.971361 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:07.971454 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:07.971814 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 03:06:07.971942 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 03:06:07.972760 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baaf8dd1595e491b93c4c010d132ccff" candidate_uuid: "278332e68f62492e90cbeb5802b32f8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" is_pre_election: true
I20260926 03:06:07.973191 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baaf8dd1595e491b93c4c010d132ccff" candidate_uuid: "278332e68f62492e90cbeb5802b32f8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b15866fca8ae4ade8630d4f36ef468d4" is_pre_election: true
I20260926 03:06:07.973348 2440 raft_consensus.cc:2454] T baaf8dd1595e491b93c4c010d132ccff P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 278332e68f62492e90cbeb5802b32f8e in term 0.
I20260926 03:06:07.973701 2301 raft_consensus.cc:2454] T baaf8dd1595e491b93c4c010d132ccff P b15866fca8ae4ade8630d4f36ef468d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 278332e68f62492e90cbeb5802b32f8e in term 0.
I20260926 03:06:07.974048 2510 leader_election.cc:302] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 278332e68f62492e90cbeb5802b32f8e, ffc2110a66234d14a58b23e4ff09ea0a; no voters:
I20260926 03:06:07.974481 2654 raft_consensus.cc:2781] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:06:07.974673 2654 raft_consensus.cc:482] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:06:07.974773 2654 raft_consensus.cc:3037] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:07.977077 2654 raft_consensus.cc:504] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:07.977905 2654 leader_election.cc:288] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e [CANDIDATE]: Term 1 election: Requested vote from peers ffc2110a66234d14a58b23e4ff09ea0a (127.0.97.2:39487), b15866fca8ae4ade8630d4f36ef468d4 (127.0.97.1:45529)
I20260926 03:06:07.978595 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baaf8dd1595e491b93c4c010d132ccff" candidate_uuid: "278332e68f62492e90cbeb5802b32f8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffc2110a66234d14a58b23e4ff09ea0a"
I20260926 03:06:07.978807 2440 raft_consensus.cc:3037] T baaf8dd1595e491b93c4c010d132ccff P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:07.978978 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baaf8dd1595e491b93c4c010d132ccff" candidate_uuid: "278332e68f62492e90cbeb5802b32f8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b15866fca8ae4ade8630d4f36ef468d4"
I20260926 03:06:07.979183 2301 raft_consensus.cc:3037] T baaf8dd1595e491b93c4c010d132ccff P b15866fca8ae4ade8630d4f36ef468d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:07.981205 2440 raft_consensus.cc:2454] T baaf8dd1595e491b93c4c010d132ccff P ffc2110a66234d14a58b23e4ff09ea0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 278332e68f62492e90cbeb5802b32f8e in term 1.
I20260926 03:06:07.981397 2301 raft_consensus.cc:2454] T baaf8dd1595e491b93c4c010d132ccff P b15866fca8ae4ade8630d4f36ef468d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 278332e68f62492e90cbeb5802b32f8e in term 1.
I20260926 03:06:07.981731 2510 leader_election.cc:302] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 278332e68f62492e90cbeb5802b32f8e, ffc2110a66234d14a58b23e4ff09ea0a; no voters:
I20260926 03:06:07.982152 2654 raft_consensus.cc:2781] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:07.982715 2654 raft_consensus.cc:686] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e [term 1 LEADER]: Becoming Leader. State: Replica: 278332e68f62492e90cbeb5802b32f8e, State: Running, Role: LEADER
I20260926 03:06:07.983266 2654 consensus_queue.cc:237] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:07.988282 2143 catalog_manager.cc:5495] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 278332e68f62492e90cbeb5802b32f8e (127.0.97.3). New cstate: current_term: 1 leader_uuid: "278332e68f62492e90cbeb5802b32f8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } health_report { overall_health: HEALTHY } } }
Sep 26 03:06:08 dist-test-slave-hmbw krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:08.101378 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:08.085383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38389 (local address 127.0.0.1:56982)
0926 03:06:08.086124 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:08.086143 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:08.086303 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:08.086634 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:08.086638 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:08.087359 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:08.087556 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:08.087576 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:08.088710 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:08.088715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:08.089676 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:08.089687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:08.090431 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:08.090440 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:08.090638 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:08.091409 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:08.091433 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:08.093882 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:08.097877 (+ 3995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:08.097900 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:08.097909 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:08.099141 (+ 1232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:08.099705 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:08.099710 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:08.099712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:08.099902 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:08.100517 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:08.100526 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:08.100682 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 03:06:08.100825 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 03:06:08.101084 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":417,"threads_started":1}
I20260926 03:06:08.101739 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:08.085383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56982 (local address 127.0.97.62:38389)
0926 03:06:08.086107 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:08.086113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:08.086218 (+ 105us) server_negotiation.cc:407] Connection header received
0926 03:06:08.086427 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:08.086431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:08.086484 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:08.086610 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:08.087747 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:08.088555 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:08.089815 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:08.090209 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:08.090411 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:08.094172 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:08.094178 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:08.094182 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:08.094217 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:08.097607 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:08.099365 (+ 1758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:08.099371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:08.099375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:08.099567 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:08.100033 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:08.100037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:08.100040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:08.100348 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:08.100491 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:08.100943 (+ 452us) server_negotiation.cc:299] Negotiation successful
0926 03:06:08.101264 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":557,"thread_start_us":366,"threads_started":1}
I20260926 03:06:08.108456 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56982:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 03:06:08.109675 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:06:08.118268 2281 tablet_service.cc:1467] Processing CreateTablet for tablet a44416d8f12046ebb289d8884d450023 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d6eff512d72e4e82bd5a7e395acff0da]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:06:08.118477 2420 tablet_service.cc:1467] Processing CreateTablet for tablet a44416d8f12046ebb289d8884d450023 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d6eff512d72e4e82bd5a7e395acff0da]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:06:08.118767 2559 tablet_service.cc:1467] Processing CreateTablet for tablet a44416d8f12046ebb289d8884d450023 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d6eff512d72e4e82bd5a7e395acff0da]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:06:08.119343 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a44416d8f12046ebb289d8884d450023. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:08.119405 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a44416d8f12046ebb289d8884d450023. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:08.119508 2559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a44416d8f12046ebb289d8884d450023. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:08.126227 2671 tablet_bootstrap.cc:492] T a44416d8f12046ebb289d8884d450023 P b15866fca8ae4ade8630d4f36ef468d4: Bootstrap starting.
I20260926 03:06:08.127238 2673 tablet_bootstrap.cc:492] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a: Bootstrap starting.
I20260926 03:06:08.127983 2672 tablet_bootstrap.cc:492] T a44416d8f12046ebb289d8884d450023 P 278332e68f62492e90cbeb5802b32f8e: Bootstrap starting.
I20260926 03:06:08.128778 2671 tablet_bootstrap.cc:654] T a44416d8f12046ebb289d8884d450023 P b15866fca8ae4ade8630d4f36ef468d4: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:08.128808 2673 tablet_bootstrap.cc:654] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:08.129534 2672 tablet_bootstrap.cc:654] T a44416d8f12046ebb289d8884d450023 P 278332e68f62492e90cbeb5802b32f8e: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:08.131634 2672 tablet_bootstrap.cc:492] T a44416d8f12046ebb289d8884d450023 P 278332e68f62492e90cbeb5802b32f8e: No bootstrap required, opened a new log
I20260926 03:06:08.131737 2673 tablet_bootstrap.cc:492] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a: No bootstrap required, opened a new log
I20260926 03:06:08.131798 2672 ts_tablet_manager.cc:1397] T a44416d8f12046ebb289d8884d450023 P 278332e68f62492e90cbeb5802b32f8e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 03:06:08.131821 2671 tablet_bootstrap.cc:492] T a44416d8f12046ebb289d8884d450023 P b15866fca8ae4ade8630d4f36ef468d4: No bootstrap required, opened a new log
I20260926 03:06:08.131889 2673 ts_tablet_manager.cc:1397] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 03:06:08.131983 2671 ts_tablet_manager.cc:1397] T a44416d8f12046ebb289d8884d450023 P b15866fca8ae4ade8630d4f36ef468d4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260926 03:06:08.132575 2672 raft_consensus.cc:348] T a44416d8f12046ebb289d8884d450023 P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:08.132666 2673 raft_consensus.cc:348] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:08.132680 2671 raft_consensus.cc:348] T a44416d8f12046ebb289d8884d450023 P b15866fca8ae4ade8630d4f36ef468d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:08.132823 2672 raft_consensus.cc:374] T a44416d8f12046ebb289d8884d450023 P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:08.132897 2673 raft_consensus.cc:374] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:08.132910 2672 raft_consensus.cc:729] T a44416d8f12046ebb289d8884d450023 P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 278332e68f62492e90cbeb5802b32f8e, State: Initialized, Role: FOLLOWER
I20260926 03:06:08.132911 2671 raft_consensus.cc:374] T a44416d8f12046ebb289d8884d450023 P b15866fca8ae4ade8630d4f36ef468d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:08.132980 2673 raft_consensus.cc:729] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffc2110a66234d14a58b23e4ff09ea0a, State: Initialized, Role: FOLLOWER
I20260926 03:06:08.132992 2671 raft_consensus.cc:729] T a44416d8f12046ebb289d8884d450023 P b15866fca8ae4ade8630d4f36ef468d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b15866fca8ae4ade8630d4f36ef468d4, State: Initialized, Role: FOLLOWER
I20260926 03:06:08.133116 2672 consensus_queue.cc:260] T a44416d8f12046ebb289d8884d450023 P 278332e68f62492e90cbeb5802b32f8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:08.133203 2673 consensus_queue.cc:260] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:08.133347 2671 consensus_queue.cc:260] T a44416d8f12046ebb289d8884d450023 P b15866fca8ae4ade8630d4f36ef468d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:08.133690 2672 ts_tablet_manager.cc:1428] T a44416d8f12046ebb289d8884d450023 P 278332e68f62492e90cbeb5802b32f8e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 03:06:08.133630 2654 tablet_replica.cc:439] T a44416d8f12046ebb289d8884d450023 P 278332e68f62492e90cbeb5802b32f8e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } } }
I20260926 03:06:08.133942 2654 tablet_replica.cc:442] T a44416d8f12046ebb289d8884d450023 P 278332e68f62492e90cbeb5802b32f8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:06:08.134294 2673 ts_tablet_manager.cc:1428] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:06:08.134786 2677 tablet_replica.cc:439] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } } }
I20260926 03:06:08.135182 2677 tablet_replica.cc:442] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:06:08.135412 2671 ts_tablet_manager.cc:1428] T a44416d8f12046ebb289d8884d450023 P b15866fca8ae4ade8630d4f36ef468d4: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260926 03:06:08.136333 2678 tablet_replica.cc:439] T a44416d8f12046ebb289d8884d450023 P b15866fca8ae4ade8630d4f36ef468d4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } } }
I20260926 03:06:08.136765 2678 tablet_replica.cc:442] T a44416d8f12046ebb289d8884d450023 P b15866fca8ae4ade8630d4f36ef468d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:06:08.187074 2677 raft_consensus.cc:482] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:06:08.187304 2677 raft_consensus.cc:504] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:08.188910 2677 leader_election.cc:288] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b15866fca8ae4ade8630d4f36ef468d4 (127.0.97.1:45529), 278332e68f62492e90cbeb5802b32f8e (127.0.97.3:34285)
I20260926 03:06:08.196339 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:08.189394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42997 (local address 127.0.97.1:45529)
0926 03:06:08.189569 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:08.189574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:08.190255 (+ 681us) server_negotiation.cc:407] Connection header received
0926 03:06:08.190444 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:08.190450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:08.190565 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:08.190649 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:08.191395 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:08.192397 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:08.194362 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:08.195300 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:08.195432 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:08.195547 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:08.195842 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 03:06:08.195986 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:06:08.196427 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:08.189251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45529 (local address 127.0.97.2:42997)
0926 03:06:08.190120 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:08.190139 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:08.190306 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:08.190824 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:08.190828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:08.190855 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:08.191239 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:08.191250 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:08.192578 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:08.192584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:08.194152 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:08.194169 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:08.195523 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:08.195535 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:08.195685 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:08.195710 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 03:06:08.195796 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 03:06:08.195944 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"mutex_wait_us":28,"spinlock_wait_cycles":16000,"thread_start_us":493,"threads_started":1}
I20260926 03:06:08.196448 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:08.190380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58555 (local address 127.0.97.3:34285)
0926 03:06:08.191110 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:08.191115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:08.191135 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:06:08.191192 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:08.191195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:08.191250 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:08.191365 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:08.191958 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:08.192696 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:08.194566 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:08.195370 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:08.195488 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:08.195600 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:08.195905 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 03:06:08.196049 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":489,"threads_started":1}
I20260926 03:06:08.196918 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:08.190243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34285 (local address 127.0.97.2:58555)
0926 03:06:08.190882 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:08.190901 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:08.191031 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:08.191408 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:08.191414 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:08.191449 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:08.191765 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:08.191782 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:08.192852 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:08.192856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:08.194355 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:08.194368 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:08.195601 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:08.195607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:08.195749 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:08.195780 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 03:06:08.195859 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 03:06:08.195993 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":19,"spinlock_wait_cycles":768,"thread_start_us":291,"threads_started":1}
I20260926 03:06:08.197134 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44416d8f12046ebb289d8884d450023" candidate_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b15866fca8ae4ade8630d4f36ef468d4" is_pre_election: true
I20260926 03:06:08.197644 2301 raft_consensus.cc:2454] T a44416d8f12046ebb289d8884d450023 P b15866fca8ae4ade8630d4f36ef468d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffc2110a66234d14a58b23e4ff09ea0a in term 0.
I20260926 03:06:08.198139 2371 leader_election.cc:302] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b15866fca8ae4ade8630d4f36ef468d4, ffc2110a66234d14a58b23e4ff09ea0a; no voters:
I20260926 03:06:08.198149 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44416d8f12046ebb289d8884d450023" candidate_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "278332e68f62492e90cbeb5802b32f8e" is_pre_election: true
I20260926 03:06:08.198590 2677 raft_consensus.cc:2781] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:06:08.198609 2579 raft_consensus.cc:2454] T a44416d8f12046ebb289d8884d450023 P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffc2110a66234d14a58b23e4ff09ea0a in term 0.
I20260926 03:06:08.198709 2677 raft_consensus.cc:482] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:06:08.198843 2677 raft_consensus.cc:3037] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:08.201062 2677 raft_consensus.cc:504] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:08.201687 2677 leader_election.cc:288] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a [CANDIDATE]: Term 1 election: Requested vote from peers b15866fca8ae4ade8630d4f36ef468d4 (127.0.97.1:45529), 278332e68f62492e90cbeb5802b32f8e (127.0.97.3:34285)
I20260926 03:06:08.202280 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44416d8f12046ebb289d8884d450023" candidate_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b15866fca8ae4ade8630d4f36ef468d4"
I20260926 03:06:08.202461 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44416d8f12046ebb289d8884d450023" candidate_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "278332e68f62492e90cbeb5802b32f8e"
I20260926 03:06:08.202589 2301 raft_consensus.cc:3037] T a44416d8f12046ebb289d8884d450023 P b15866fca8ae4ade8630d4f36ef468d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:08.202708 2579 raft_consensus.cc:3037] T a44416d8f12046ebb289d8884d450023 P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:08.206125 2579 raft_consensus.cc:2454] T a44416d8f12046ebb289d8884d450023 P 278332e68f62492e90cbeb5802b32f8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffc2110a66234d14a58b23e4ff09ea0a in term 1.
I20260926 03:06:08.206113 2301 raft_consensus.cc:2454] T a44416d8f12046ebb289d8884d450023 P b15866fca8ae4ade8630d4f36ef468d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffc2110a66234d14a58b23e4ff09ea0a in term 1.
I20260926 03:06:08.206630 2371 leader_election.cc:302] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b15866fca8ae4ade8630d4f36ef468d4, ffc2110a66234d14a58b23e4ff09ea0a; no voters:
I20260926 03:06:08.206970 2677 raft_consensus.cc:2781] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:08.207557 2677 raft_consensus.cc:686] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a [term 1 LEADER]: Becoming Leader. State: Replica: ffc2110a66234d14a58b23e4ff09ea0a, State: Running, Role: LEADER
I20260926 03:06:08.208057 2677 consensus_queue.cc:237] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:08.211506 2682 tablet_replica.cc:439] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a: TxnStatusTablet state changed. Reason: New leader ffc2110a66234d14a58b23e4ff09ea0a. Latest consensus state: current_term: 1 leader_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } } }
I20260926 03:06:08.211882 2682 tablet_replica.cc:442] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:06:08.212589 2684 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 03:06:08.212867 2684 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 03:06:08.213388 2142 catalog_manager.cc:5495] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a reported cstate change: term changed from 0 to 1, leader changed from <none> to ffc2110a66234d14a58b23e4ff09ea0a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } health_report { overall_health: UNKNOWN } } }
I20260926 03:06:08.240661 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:08.234764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39742 (local address 127.0.97.2:39487)
0926 03:06:08.234977 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:08.234984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:08.235136 (+ 152us) server_negotiation.cc:407] Connection header received
0926 03:06:08.235527 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:08.235531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:08.235617 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:08.235739 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:06:08.236075 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:08.236862 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:08.238098 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:08.238587 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:08.238707 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:08.239329 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:06:08.239956 (+ 627us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:06:08.240080 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:08.240336 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 03:06:08.240409 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 03:06:08.240697 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:08.234783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39487 (local address 127.0.0.1:39742)
0926 03:06:08.235048 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:08.235064 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:08.235212 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:08.235771 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:08.235775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:08.235787 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:06:08.235954 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:08.235962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:08.236984 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:08.236990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:08.237958 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:08.237969 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:08.238784 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:08.238792 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:08.239020 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:08.239104 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:06:08.240100 (+ 996us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:06:08.240135 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 03:06:08.240235 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 03:06:08.240403 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 03:06:08.250365 2301 raft_consensus.cc:1261] T a44416d8f12046ebb289d8884d450023 P b15866fca8ae4ade8630d4f36ef468d4 [term 1 FOLLOWER]: Refusing update from remote peer ffc2110a66234d14a58b23e4ff09ea0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:08.250334 2579 raft_consensus.cc:1261] T a44416d8f12046ebb289d8884d450023 P 278332e68f62492e90cbeb5802b32f8e [term 1 FOLLOWER]: Refusing update from remote peer ffc2110a66234d14a58b23e4ff09ea0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:08.251382 2682 consensus_queue.cc:1035] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:08.251837 2677 consensus_queue.cc:1035] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:08.263626 2654 tablet_replica.cc:439] T a44416d8f12046ebb289d8884d450023 P 278332e68f62492e90cbeb5802b32f8e: TxnStatusTablet state changed. Reason: New leader ffc2110a66234d14a58b23e4ff09ea0a. Latest consensus state: current_term: 1 leader_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } } }
I20260926 03:06:08.264003 2654 tablet_replica.cc:442] T a44416d8f12046ebb289d8884d450023 P 278332e68f62492e90cbeb5802b32f8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:06:08.267249 2687 mvcc.cc:205] Tried to move back new op lower bound from 7333445501942378496 to 7333445501783474176. Current Snapshot: MvccSnapshot[applied={T|T < 7333445501942378496}]
I20260926 03:06:08.278390 2677 tablet_replica.cc:439] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } } }
I20260926 03:06:08.278750 2677 tablet_replica.cc:442] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:06:08.287557 2654 tablet_replica.cc:439] T a44416d8f12046ebb289d8884d450023 P 278332e68f62492e90cbeb5802b32f8e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } } }
I20260926 03:06:08.288007 2654 tablet_replica.cc:442] T a44416d8f12046ebb289d8884d450023 P 278332e68f62492e90cbeb5802b32f8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:06:08.291040 2678 tablet_replica.cc:439] T a44416d8f12046ebb289d8884d450023 P b15866fca8ae4ade8630d4f36ef468d4: TxnStatusTablet state changed. Reason: New leader ffc2110a66234d14a58b23e4ff09ea0a. Latest consensus state: current_term: 1 leader_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } } }
I20260926 03:06:08.291360 2678 tablet_replica.cc:442] T a44416d8f12046ebb289d8884d450023 P b15866fca8ae4ade8630d4f36ef468d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:06:08.295308 2682 tablet_replica.cc:439] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } } }
I20260926 03:06:08.295598 2682 tablet_replica.cc:442] T a44416d8f12046ebb289d8884d450023 P ffc2110a66234d14a58b23e4ff09ea0a: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:06:08.308239 2678 tablet_replica.cc:439] T a44416d8f12046ebb289d8884d450023 P b15866fca8ae4ade8630d4f36ef468d4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } } }
I20260926 03:06:08.308521 2678 tablet_replica.cc:442] T a44416d8f12046ebb289d8884d450023 P b15866fca8ae4ade8630d4f36ef468d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:06:08.317660 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:08.309748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34768 (local address 127.0.97.3:34285)
0926 03:06:08.309919 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:08.309926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:08.310103 (+ 177us) server_negotiation.cc:407] Connection header received
0926 03:06:08.310421 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:08.310427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:08.310579 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:08.310708 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:06:08.311460 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:08.312531 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:08.314700 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:08.315370 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:08.315506 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:08.316026 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:06:08.316786 (+ 760us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:06:08.316943 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:08.317168 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 03:06:08.317270 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 03:06:08.336993 2700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:08.330338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39487 (local address 127.0.97.3:52071)
0926 03:06:08.331080 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:08.331096 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:08.331225 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:08.331688 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:08.331692 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:08.331703 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:06:08.332036 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:08.332046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:08.333530 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:08.333535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:08.334764 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:08.334778 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:08.335817 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:08.335823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:08.335950 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:08.336015 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:06:08.336492 (+ 477us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:06:08.336504 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 03:06:08.336584 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 03:06:08.336699 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":430,"threads_started":1}
I20260926 03:06:08.336990 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:08.330372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52071 (local address 127.0.97.2:39487)
0926 03:06:08.330590 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:08.330596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:08.331187 (+ 591us) server_negotiation.cc:407] Connection header received
0926 03:06:08.331352 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:08.331358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:08.331452 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:08.331563 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:06:08.332200 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:08.333345 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:08.334918 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:08.335594 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:08.335722 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:08.336235 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:06:08.336370 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:06:08.336470 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:08.336617 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 03:06:08.336704 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:06:08.348490 2301 raft_consensus.cc:1261] T baaf8dd1595e491b93c4c010d132ccff P b15866fca8ae4ade8630d4f36ef468d4 [term 1 FOLLOWER]: Refusing update from remote peer 278332e68f62492e90cbeb5802b32f8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:08.348795 2440 raft_consensus.cc:1261] T baaf8dd1595e491b93c4c010d132ccff P ffc2110a66234d14a58b23e4ff09ea0a [term 1 FOLLOWER]: Refusing update from remote peer 278332e68f62492e90cbeb5802b32f8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:08.349176 2659 consensus_queue.cc:1035] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:08.349553 2654 consensus_queue.cc:1035] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:08.403613 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:08.389613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34285 (local address 127.0.97.2:36537)
0926 03:06:08.396424 (+ 6811us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:08.396442 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:08.396580 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:08.396985 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:08.396990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:08.397003 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:06:08.397443 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:08.397457 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:08.399100 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:08.399106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:08.400957 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:08.400971 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:08.402066 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:08.402073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:08.402295 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:08.402417 (+ 122us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:06:08.402912 (+ 495us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:06:08.402923 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 03:06:08.402996 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 03:06:08.403147 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6578,"thread_start_us":6386,"threads_started":1}
I20260926 03:06:08.403527 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:08.390275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36537 (local address 127.0.97.3:34285)
0926 03:06:08.391705 (+ 1430us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:08.391711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:08.396531 (+ 4820us) server_negotiation.cc:407] Connection header received
0926 03:06:08.396699 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:08.396705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:08.396798 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:08.396896 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:06:08.397630 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:08.398957 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:08.401141 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:08.401861 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:08.401995 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:08.402674 (+ 679us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:06:08.402816 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:06:08.402906 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:08.403014 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 03:06:08.403097 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1294}
I20260926 03:06:08.453627 2141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:06:08.453984 2141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:06:08.458477 2141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b80281939834119bff32889ec34ea5b: Sending DeleteTablet for 3 replicas of tablet baaf8dd1595e491b93c4c010d132ccff
I20260926 03:06:08.459868 2559 tablet_service.cc:1514] Processing DeleteTablet for tablet baaf8dd1595e491b93c4c010d132ccff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34764
I20260926 03:06:08.459951 2281 tablet_service.cc:1514] Processing DeleteTablet for tablet baaf8dd1595e491b93c4c010d132ccff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60564
I20260926 03:06:08.460916 2420 tablet_service.cc:1514] Processing DeleteTablet for tablet baaf8dd1595e491b93c4c010d132ccff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39730
I20260926 03:06:08.469447 2141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:06:08.471839 2141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:06:08.474104 2708 tablet_replica.cc:330] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e: stopping tablet replica
I20260926 03:06:08.474700 2708 raft_consensus.cc:2229] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:06:08.475409 2708 raft_consensus.cc:2258] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:08.477640 2708 ts_tablet_manager.cc:1905] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:08.484412 2280 tablet_service.cc:1467] Processing CreateTablet for tablet e784c23cbe1640938f3c789709c90339 (DEFAULT_TABLE table=test-table [id=a7eb6449b64d4184b99890158fdab561]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:08.485023 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e784c23cbe1640938f3c789709c90339. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:08.485138 2559 tablet_service.cc:1467] Processing CreateTablet for tablet e784c23cbe1640938f3c789709c90339 (DEFAULT_TABLE table=test-table [id=a7eb6449b64d4184b99890158fdab561]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:08.485559 2559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e784c23cbe1640938f3c789709c90339. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:08.491137 2419 tablet_service.cc:1467] Processing CreateTablet for tablet e784c23cbe1640938f3c789709c90339 (DEFAULT_TABLE table=test-table [id=a7eb6449b64d4184b99890158fdab561]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:08.491717 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e784c23cbe1640938f3c789709c90339. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:08.493045 2708 ts_tablet_manager.cc:1918] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:06:08.493240 2708 log.cc:1196] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-2/wal/wals/baaf8dd1595e491b93c4c010d132ccff
I20260926 03:06:08.493760 2708 ts_tablet_manager.cc:1939] T baaf8dd1595e491b93c4c010d132ccff P 278332e68f62492e90cbeb5802b32f8e: Deleting consensus metadata
I20260926 03:06:08.495012 2672 tablet_bootstrap.cc:492] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e: Bootstrap starting.
I20260926 03:06:08.497077 2672 tablet_bootstrap.cc:654] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:08.497392 2128 catalog_manager.cc:4841] TS 278332e68f62492e90cbeb5802b32f8e (127.0.97.3:34285): tablet baaf8dd1595e491b93c4c010d132ccff (table test-table [id=29d311c7122e457fb1309720bcd47356]) successfully deleted
I20260926 03:06:08.501376 2673 tablet_bootstrap.cc:492] T e784c23cbe1640938f3c789709c90339 P ffc2110a66234d14a58b23e4ff09ea0a: Bootstrap starting.
I20260926 03:06:08.502341 2671 tablet_bootstrap.cc:492] T e784c23cbe1640938f3c789709c90339 P b15866fca8ae4ade8630d4f36ef468d4: Bootstrap starting.
I20260926 03:06:08.503190 2673 tablet_bootstrap.cc:654] T e784c23cbe1640938f3c789709c90339 P ffc2110a66234d14a58b23e4ff09ea0a: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:08.503345 2710 tablet_replica.cc:330] T baaf8dd1595e491b93c4c010d132ccff P ffc2110a66234d14a58b23e4ff09ea0a: stopping tablet replica
I20260926 03:06:08.504005 2671 tablet_bootstrap.cc:654] T e784c23cbe1640938f3c789709c90339 P b15866fca8ae4ade8630d4f36ef468d4: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:08.504410 2710 raft_consensus.cc:2229] T baaf8dd1595e491b93c4c010d132ccff P ffc2110a66234d14a58b23e4ff09ea0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:06:08.504571 2709 tablet_replica.cc:330] T baaf8dd1595e491b93c4c010d132ccff P b15866fca8ae4ade8630d4f36ef468d4: stopping tablet replica
I20260926 03:06:08.504911 2710 raft_consensus.cc:2258] T baaf8dd1595e491b93c4c010d132ccff P ffc2110a66234d14a58b23e4ff09ea0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:08.505393 2709 raft_consensus.cc:2229] T baaf8dd1595e491b93c4c010d132ccff P b15866fca8ae4ade8630d4f36ef468d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:06:08.506067 2709 raft_consensus.cc:2258] T baaf8dd1595e491b93c4c010d132ccff P b15866fca8ae4ade8630d4f36ef468d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:08.509702 2709 ts_tablet_manager.cc:1905] T baaf8dd1595e491b93c4c010d132ccff P b15866fca8ae4ade8630d4f36ef468d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:08.510865 2710 ts_tablet_manager.cc:1905] T baaf8dd1595e491b93c4c010d132ccff P ffc2110a66234d14a58b23e4ff09ea0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:08.511740 2673 tablet_bootstrap.cc:492] T e784c23cbe1640938f3c789709c90339 P ffc2110a66234d14a58b23e4ff09ea0a: No bootstrap required, opened a new log
I20260926 03:06:08.512068 2673 ts_tablet_manager.cc:1397] T e784c23cbe1640938f3c789709c90339 P ffc2110a66234d14a58b23e4ff09ea0a: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.003s
I20260926 03:06:08.512918 2673 raft_consensus.cc:348] T e784c23cbe1640938f3c789709c90339 P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:08.513193 2673 raft_consensus.cc:374] T e784c23cbe1640938f3c789709c90339 P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:08.513269 2673 raft_consensus.cc:729] T e784c23cbe1640938f3c789709c90339 P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffc2110a66234d14a58b23e4ff09ea0a, State: Initialized, Role: FOLLOWER
I20260926 03:06:08.513741 2673 consensus_queue.cc:260] T e784c23cbe1640938f3c789709c90339 P ffc2110a66234d14a58b23e4ff09ea0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:08.517201 2673 ts_tablet_manager.cc:1428] T e784c23cbe1640938f3c789709c90339 P ffc2110a66234d14a58b23e4ff09ea0a: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260926 03:06:08.521525 2671 tablet_bootstrap.cc:492] T e784c23cbe1640938f3c789709c90339 P b15866fca8ae4ade8630d4f36ef468d4: No bootstrap required, opened a new log
I20260926 03:06:08.521744 2671 ts_tablet_manager.cc:1397] T e784c23cbe1640938f3c789709c90339 P b15866fca8ae4ade8630d4f36ef468d4: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.001s
I20260926 03:06:08.522523 2671 raft_consensus.cc:348] T e784c23cbe1640938f3c789709c90339 P b15866fca8ae4ade8630d4f36ef468d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:08.522804 2671 raft_consensus.cc:374] T e784c23cbe1640938f3c789709c90339 P b15866fca8ae4ade8630d4f36ef468d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:08.522863 2671 raft_consensus.cc:729] T e784c23cbe1640938f3c789709c90339 P b15866fca8ae4ade8630d4f36ef468d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b15866fca8ae4ade8630d4f36ef468d4, State: Initialized, Role: FOLLOWER
I20260926 03:06:08.523083 2671 consensus_queue.cc:260] T e784c23cbe1640938f3c789709c90339 P b15866fca8ae4ade8630d4f36ef468d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:08.523610 2671 ts_tablet_manager.cc:1428] T e784c23cbe1640938f3c789709c90339 P b15866fca8ae4ade8630d4f36ef468d4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 03:06:08.531824 2709 ts_tablet_manager.cc:1918] T baaf8dd1595e491b93c4c010d132ccff P b15866fca8ae4ade8630d4f36ef468d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:06:08.538273 2710 ts_tablet_manager.cc:1918] T baaf8dd1595e491b93c4c010d132ccff P ffc2110a66234d14a58b23e4ff09ea0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:06:08.538738 2710 log.cc:1196] T baaf8dd1595e491b93c4c010d132ccff P ffc2110a66234d14a58b23e4ff09ea0a: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-1/wal/wals/baaf8dd1595e491b93c4c010d132ccff
I20260926 03:06:08.538967 2709 log.cc:1196] T baaf8dd1595e491b93c4c010d132ccff P b15866fca8ae4ade8630d4f36ef468d4: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-0/wal/wals/baaf8dd1595e491b93c4c010d132ccff
I20260926 03:06:08.539023 2672 tablet_bootstrap.cc:492] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e: No bootstrap required, opened a new log
I20260926 03:06:08.539252 2672 ts_tablet_manager.cc:1397] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.017s
I20260926 03:06:08.539872 2709 ts_tablet_manager.cc:1939] T baaf8dd1595e491b93c4c010d132ccff P b15866fca8ae4ade8630d4f36ef468d4: Deleting consensus metadata
I20260926 03:06:08.540050 2672 raft_consensus.cc:348] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:08.540351 2672 raft_consensus.cc:374] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:08.540439 2672 raft_consensus.cc:729] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 278332e68f62492e90cbeb5802b32f8e, State: Initialized, Role: FOLLOWER
I20260926 03:06:08.540462 2710 ts_tablet_manager.cc:1939] T baaf8dd1595e491b93c4c010d132ccff P ffc2110a66234d14a58b23e4ff09ea0a: Deleting consensus metadata
I20260926 03:06:08.540759 2672 consensus_queue.cc:260] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:08.542790 2128 catalog_manager.cc:4841] TS b15866fca8ae4ade8630d4f36ef468d4 (127.0.97.1:45529): tablet baaf8dd1595e491b93c4c010d132ccff (table test-table [id=29d311c7122e457fb1309720bcd47356]) successfully deleted
I20260926 03:06:08.544224 2128 catalog_manager.cc:4841] TS ffc2110a66234d14a58b23e4ff09ea0a (127.0.97.2:39487): tablet baaf8dd1595e491b93c4c010d132ccff (table test-table [id=29d311c7122e457fb1309720bcd47356]) successfully deleted
I20260926 03:06:08.546203 2672 ts_tablet_manager.cc:1428] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260926 03:06:08.614651 2654 raft_consensus.cc:482] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:06:08.614879 2654 raft_consensus.cc:504] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:08.615691 2654 leader_election.cc:288] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b15866fca8ae4ade8630d4f36ef468d4 (127.0.97.1:45529), ffc2110a66234d14a58b23e4ff09ea0a (127.0.97.2:39487)
I20260926 03:06:08.616365 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e784c23cbe1640938f3c789709c90339" candidate_uuid: "278332e68f62492e90cbeb5802b32f8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b15866fca8ae4ade8630d4f36ef468d4" is_pre_election: true
I20260926 03:06:08.616600 2301 raft_consensus.cc:2454] T e784c23cbe1640938f3c789709c90339 P b15866fca8ae4ade8630d4f36ef468d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 278332e68f62492e90cbeb5802b32f8e in term 0.
I20260926 03:06:08.616505 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e784c23cbe1640938f3c789709c90339" candidate_uuid: "278332e68f62492e90cbeb5802b32f8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" is_pre_election: true
I20260926 03:06:08.616817 2440 raft_consensus.cc:2454] T e784c23cbe1640938f3c789709c90339 P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 278332e68f62492e90cbeb5802b32f8e in term 0.
I20260926 03:06:08.617475 2510 leader_election.cc:302] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 278332e68f62492e90cbeb5802b32f8e, ffc2110a66234d14a58b23e4ff09ea0a; no voters:
I20260926 03:06:08.617830 2654 raft_consensus.cc:2781] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:06:08.617970 2654 raft_consensus.cc:482] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:06:08.618059 2654 raft_consensus.cc:3037] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:08.621394 2654 raft_consensus.cc:504] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:08.622227 2654 leader_election.cc:288] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e [CANDIDATE]: Term 1 election: Requested vote from peers b15866fca8ae4ade8630d4f36ef468d4 (127.0.97.1:45529), ffc2110a66234d14a58b23e4ff09ea0a (127.0.97.2:39487)
I20260926 03:06:08.623036 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e784c23cbe1640938f3c789709c90339" candidate_uuid: "278332e68f62492e90cbeb5802b32f8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b15866fca8ae4ade8630d4f36ef468d4"
I20260926 03:06:08.623062 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e784c23cbe1640938f3c789709c90339" candidate_uuid: "278332e68f62492e90cbeb5802b32f8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffc2110a66234d14a58b23e4ff09ea0a"
I20260926 03:06:08.623284 2440 raft_consensus.cc:3037] T e784c23cbe1640938f3c789709c90339 P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:08.623282 2301 raft_consensus.cc:3037] T e784c23cbe1640938f3c789709c90339 P b15866fca8ae4ade8630d4f36ef468d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:08.626331 2301 raft_consensus.cc:2454] T e784c23cbe1640938f3c789709c90339 P b15866fca8ae4ade8630d4f36ef468d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 278332e68f62492e90cbeb5802b32f8e in term 1.
I20260926 03:06:08.627173 2440 raft_consensus.cc:2454] T e784c23cbe1640938f3c789709c90339 P ffc2110a66234d14a58b23e4ff09ea0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 278332e68f62492e90cbeb5802b32f8e in term 1.
I20260926 03:06:08.627192 2510 leader_election.cc:302] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 278332e68f62492e90cbeb5802b32f8e, b15866fca8ae4ade8630d4f36ef468d4; no voters:
I20260926 03:06:08.627712 2654 raft_consensus.cc:2781] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:08.627897 2654 raft_consensus.cc:686] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e [term 1 LEADER]: Becoming Leader. State: Replica: 278332e68f62492e90cbeb5802b32f8e, State: Running, Role: LEADER
I20260926 03:06:08.628213 2654 consensus_queue.cc:237] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:08.636920 2142 catalog_manager.cc:5495] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 278332e68f62492e90cbeb5802b32f8e (127.0.97.3). New cstate: current_term: 1 leader_uuid: "278332e68f62492e90cbeb5802b32f8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } health_report { overall_health: HEALTHY } } }
I20260926 03:06:08.695956 2300 raft_consensus.cc:1261] T e784c23cbe1640938f3c789709c90339 P b15866fca8ae4ade8630d4f36ef468d4 [term 1 FOLLOWER]: Refusing update from remote peer 278332e68f62492e90cbeb5802b32f8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:08.696897 2440 raft_consensus.cc:1261] T e784c23cbe1640938f3c789709c90339 P ffc2110a66234d14a58b23e4ff09ea0a [term 1 FOLLOWER]: Refusing update from remote peer 278332e68f62492e90cbeb5802b32f8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:08.698015 2659 consensus_queue.cc:1035] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:08.698398 2654 consensus_queue.cc:1035] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:08.706187 2688 mvcc.cc:205] Tried to move back new op lower bound from 7333445503771136000 to 7333445503503831040. Current Snapshot: MvccSnapshot[applied={T|T < 7333445503771136000}]
I20260926 03:06:08.714617 2689 mvcc.cc:205] Tried to move back new op lower bound from 7333445503771136000 to 7333445503503831040. Current Snapshot: MvccSnapshot[applied={T|T < 7333445503771136000}]
W20260926 03:06:08.731667 2688 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 03:06:10.336557 2141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:06:10.336788 2141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:06:10.341378 2141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b80281939834119bff32889ec34ea5b: Sending DeleteTablet for 3 replicas of tablet e784c23cbe1640938f3c789709c90339
I20260926 03:06:10.342309 2281 tablet_service.cc:1514] Processing DeleteTablet for tablet e784c23cbe1640938f3c789709c90339 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60564
I20260926 03:06:10.342319 2419 tablet_service.cc:1514] Processing DeleteTablet for tablet e784c23cbe1640938f3c789709c90339 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39730
I20260926 03:06:10.342556 2559 tablet_service.cc:1514] Processing DeleteTablet for tablet e784c23cbe1640938f3c789709c90339 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34764
I20260926 03:06:10.343267 2738 tablet_replica.cc:330] T e784c23cbe1640938f3c789709c90339 P b15866fca8ae4ade8630d4f36ef468d4: stopping tablet replica
I20260926 03:06:10.343487 2738 raft_consensus.cc:2229] T e784c23cbe1640938f3c789709c90339 P b15866fca8ae4ade8630d4f36ef468d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:06:10.343746 2738 raft_consensus.cc:2258] T e784c23cbe1640938f3c789709c90339 P b15866fca8ae4ade8630d4f36ef468d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:10.347280 2739 tablet_replica.cc:330] T e784c23cbe1640938f3c789709c90339 P ffc2110a66234d14a58b23e4ff09ea0a: stopping tablet replica
I20260926 03:06:10.347390 2738 ts_tablet_manager.cc:1905] T e784c23cbe1640938f3c789709c90339 P b15866fca8ae4ade8630d4f36ef468d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:10.348009 2739 raft_consensus.cc:2229] T e784c23cbe1640938f3c789709c90339 P ffc2110a66234d14a58b23e4ff09ea0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:06:10.348372 2739 raft_consensus.cc:2258] T e784c23cbe1640938f3c789709c90339 P ffc2110a66234d14a58b23e4ff09ea0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:10.349941 2739 ts_tablet_manager.cc:1905] T e784c23cbe1640938f3c789709c90339 P ffc2110a66234d14a58b23e4ff09ea0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:10.355190 2738 ts_tablet_manager.cc:1918] T e784c23cbe1640938f3c789709c90339 P b15866fca8ae4ade8630d4f36ef468d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:06:10.355376 2738 log.cc:1196] T e784c23cbe1640938f3c789709c90339 P b15866fca8ae4ade8630d4f36ef468d4: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-0/wal/wals/e784c23cbe1640938f3c789709c90339
I20260926 03:06:10.355883 2738 ts_tablet_manager.cc:1939] T e784c23cbe1640938f3c789709c90339 P b15866fca8ae4ade8630d4f36ef468d4: Deleting consensus metadata
I20260926 03:06:10.357280 2740 tablet_replica.cc:330] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e: stopping tablet replica
I20260926 03:06:10.357313 2128 catalog_manager.cc:4841] TS b15866fca8ae4ade8630d4f36ef468d4 (127.0.97.1:45529): tablet e784c23cbe1640938f3c789709c90339 (table test-table [id=a7eb6449b64d4184b99890158fdab561]) successfully deleted
I20260926 03:06:10.357496 2740 raft_consensus.cc:2229] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:06:10.357484 2739 ts_tablet_manager.cc:1918] T e784c23cbe1640938f3c789709c90339 P ffc2110a66234d14a58b23e4ff09ea0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:06:10.357694 2739 log.cc:1196] T e784c23cbe1640938f3c789709c90339 P ffc2110a66234d14a58b23e4ff09ea0a: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-1/wal/wals/e784c23cbe1640938f3c789709c90339
I20260926 03:06:10.357873 2740 raft_consensus.cc:2258] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:10.358112 2739 ts_tablet_manager.cc:1939] T e784c23cbe1640938f3c789709c90339 P ffc2110a66234d14a58b23e4ff09ea0a: Deleting consensus metadata
I20260926 03:06:10.358788 2740 ts_tablet_manager.cc:1905] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:10.359539 2128 catalog_manager.cc:4841] TS ffc2110a66234d14a58b23e4ff09ea0a (127.0.97.2:39487): tablet e784c23cbe1640938f3c789709c90339 (table test-table [id=a7eb6449b64d4184b99890158fdab561]) successfully deleted
I20260926 03:06:10.364183 2740 ts_tablet_manager.cc:1918] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:06:10.364346 2740 log.cc:1196] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-2/wal/wals/e784c23cbe1640938f3c789709c90339
I20260926 03:06:10.364784 2740 ts_tablet_manager.cc:1939] T e784c23cbe1640938f3c789709c90339 P 278332e68f62492e90cbeb5802b32f8e: Deleting consensus metadata
I20260926 03:06:10.366118 2128 catalog_manager.cc:4841] TS 278332e68f62492e90cbeb5802b32f8e (127.0.97.3:34285): tablet e784c23cbe1640938f3c789709c90339 (table test-table [id=a7eb6449b64d4184b99890158fdab561]) successfully deleted
Sep 26 03:06:10 dist-test-slave-hmbw krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391970, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 03:06:10 dist-test-slave-hmbw krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391970, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:10.403476 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:10.384298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56984 (local address 127.0.97.62:38389)
0926 03:06:10.387281 (+ 2983us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:10.387288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:10.387700 (+ 412us) server_negotiation.cc:407] Connection header received
0926 03:06:10.387883 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:10.387887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:10.387961 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:10.388070 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:10.389785 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:10.390716 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:10.392193 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:10.392619 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:10.392743 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:10.397774 (+ 5031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:10.397781 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:10.397784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:10.397824 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:10.400608 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:10.401423 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:10.401428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:10.401430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:10.401605 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:10.401979 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:10.401984 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:10.401988 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:10.402304 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:10.402562 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:10.402891 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 03:06:10.403110 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2747,"thread_start_us":2547,"threads_started":1}
I20260926 03:06:10.408577 2141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:06:10.409162 2141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:06:10.416534 2281 tablet_service.cc:1467] Processing CreateTablet for tablet 6f781701dae34974ac9c84b575565ab9 (DEFAULT_TABLE table=test-table [id=4b683ca0e12c40fe95fd144a17563dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:10.416687 2419 tablet_service.cc:1467] Processing CreateTablet for tablet 6f781701dae34974ac9c84b575565ab9 (DEFAULT_TABLE table=test-table [id=4b683ca0e12c40fe95fd144a17563dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:10.416765 2559 tablet_service.cc:1467] Processing CreateTablet for tablet 6f781701dae34974ac9c84b575565ab9 (DEFAULT_TABLE table=test-table [id=4b683ca0e12c40fe95fd144a17563dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:10.417052 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f781701dae34974ac9c84b575565ab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:10.417052 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f781701dae34974ac9c84b575565ab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:10.417194 2559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f781701dae34974ac9c84b575565ab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:10.424700 2750 tablet_bootstrap.cc:492] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a: Bootstrap starting.
I20260926 03:06:10.424904 2751 tablet_bootstrap.cc:492] T 6f781701dae34974ac9c84b575565ab9 P b15866fca8ae4ade8630d4f36ef468d4: Bootstrap starting.
I20260926 03:06:10.425572 2749 tablet_bootstrap.cc:492] T 6f781701dae34974ac9c84b575565ab9 P 278332e68f62492e90cbeb5802b32f8e: Bootstrap starting.
I20260926 03:06:10.426941 2751 tablet_bootstrap.cc:654] T 6f781701dae34974ac9c84b575565ab9 P b15866fca8ae4ade8630d4f36ef468d4: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:10.427385 2749 tablet_bootstrap.cc:654] T 6f781701dae34974ac9c84b575565ab9 P 278332e68f62492e90cbeb5802b32f8e: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:10.429936 2750 tablet_bootstrap.cc:654] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:10.430541 2749 tablet_bootstrap.cc:492] T 6f781701dae34974ac9c84b575565ab9 P 278332e68f62492e90cbeb5802b32f8e: No bootstrap required, opened a new log
I20260926 03:06:10.430718 2749 ts_tablet_manager.cc:1397] T 6f781701dae34974ac9c84b575565ab9 P 278332e68f62492e90cbeb5802b32f8e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 03:06:10.431252 2751 tablet_bootstrap.cc:492] T 6f781701dae34974ac9c84b575565ab9 P b15866fca8ae4ade8630d4f36ef468d4: No bootstrap required, opened a new log
I20260926 03:06:10.431437 2751 ts_tablet_manager.cc:1397] T 6f781701dae34974ac9c84b575565ab9 P b15866fca8ae4ade8630d4f36ef468d4: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 03:06:10.431522 2749 raft_consensus.cc:348] T 6f781701dae34974ac9c84b575565ab9 P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:10.431881 2749 raft_consensus.cc:374] T 6f781701dae34974ac9c84b575565ab9 P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:10.431991 2749 raft_consensus.cc:729] T 6f781701dae34974ac9c84b575565ab9 P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 278332e68f62492e90cbeb5802b32f8e, State: Initialized, Role: FOLLOWER
I20260926 03:06:10.432247 2749 consensus_queue.cc:260] T 6f781701dae34974ac9c84b575565ab9 P 278332e68f62492e90cbeb5802b32f8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:10.432852 2749 ts_tablet_manager.cc:1428] T 6f781701dae34974ac9c84b575565ab9 P 278332e68f62492e90cbeb5802b32f8e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 03:06:10.432700 2751 raft_consensus.cc:348] T 6f781701dae34974ac9c84b575565ab9 P b15866fca8ae4ade8630d4f36ef468d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:10.433017 2751 raft_consensus.cc:374] T 6f781701dae34974ac9c84b575565ab9 P b15866fca8ae4ade8630d4f36ef468d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:10.433151 2751 raft_consensus.cc:729] T 6f781701dae34974ac9c84b575565ab9 P b15866fca8ae4ade8630d4f36ef468d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b15866fca8ae4ade8630d4f36ef468d4, State: Initialized, Role: FOLLOWER
I20260926 03:06:10.433454 2751 consensus_queue.cc:260] T 6f781701dae34974ac9c84b575565ab9 P b15866fca8ae4ade8630d4f36ef468d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:10.434376 2751 ts_tablet_manager.cc:1428] T 6f781701dae34974ac9c84b575565ab9 P b15866fca8ae4ade8630d4f36ef468d4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 03:06:10.434620 2750 tablet_bootstrap.cc:492] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a: No bootstrap required, opened a new log
I20260926 03:06:10.434758 2750 ts_tablet_manager.cc:1397] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.005s
I20260926 03:06:10.435530 2750 raft_consensus.cc:348] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:10.435786 2750 raft_consensus.cc:374] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:10.435904 2750 raft_consensus.cc:729] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffc2110a66234d14a58b23e4ff09ea0a, State: Initialized, Role: FOLLOWER
I20260926 03:06:10.436241 2750 consensus_queue.cc:260] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:10.436926 2750 ts_tablet_manager.cc:1428] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 03:06:10.456875 2677 raft_consensus.cc:482] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:06:10.457116 2677 raft_consensus.cc:504] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:10.457901 2677 leader_election.cc:288] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b15866fca8ae4ade8630d4f36ef468d4 (127.0.97.1:45529), 278332e68f62492e90cbeb5802b32f8e (127.0.97.3:34285)
I20260926 03:06:10.458619 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f781701dae34974ac9c84b575565ab9" candidate_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b15866fca8ae4ade8630d4f36ef468d4" is_pre_election: true
I20260926 03:06:10.458621 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f781701dae34974ac9c84b575565ab9" candidate_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "278332e68f62492e90cbeb5802b32f8e" is_pre_election: true
I20260926 03:06:10.458865 2579 raft_consensus.cc:2454] T 6f781701dae34974ac9c84b575565ab9 P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffc2110a66234d14a58b23e4ff09ea0a in term 0.
I20260926 03:06:10.458886 2301 raft_consensus.cc:2454] T 6f781701dae34974ac9c84b575565ab9 P b15866fca8ae4ade8630d4f36ef468d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffc2110a66234d14a58b23e4ff09ea0a in term 0.
I20260926 03:06:10.459326 2371 leader_election.cc:302] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 278332e68f62492e90cbeb5802b32f8e, ffc2110a66234d14a58b23e4ff09ea0a; no voters:
I20260926 03:06:10.459626 2677 raft_consensus.cc:2781] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:06:10.459743 2677 raft_consensus.cc:482] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:06:10.459827 2677 raft_consensus.cc:3037] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:10.462298 2677 raft_consensus.cc:504] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:10.462913 2677 leader_election.cc:288] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a [CANDIDATE]: Term 1 election: Requested vote from peers b15866fca8ae4ade8630d4f36ef468d4 (127.0.97.1:45529), 278332e68f62492e90cbeb5802b32f8e (127.0.97.3:34285)
I20260926 03:06:10.463508 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f781701dae34974ac9c84b575565ab9" candidate_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "278332e68f62492e90cbeb5802b32f8e"
I20260926 03:06:10.463549 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f781701dae34974ac9c84b575565ab9" candidate_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b15866fca8ae4ade8630d4f36ef468d4"
I20260926 03:06:10.463722 2579 raft_consensus.cc:3037] T 6f781701dae34974ac9c84b575565ab9 P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:10.463768 2301 raft_consensus.cc:3037] T 6f781701dae34974ac9c84b575565ab9 P b15866fca8ae4ade8630d4f36ef468d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:10.478554 2579 raft_consensus.cc:2454] T 6f781701dae34974ac9c84b575565ab9 P 278332e68f62492e90cbeb5802b32f8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffc2110a66234d14a58b23e4ff09ea0a in term 1.
I20260926 03:06:10.478622 2301 raft_consensus.cc:2454] T 6f781701dae34974ac9c84b575565ab9 P b15866fca8ae4ade8630d4f36ef468d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffc2110a66234d14a58b23e4ff09ea0a in term 1.
I20260926 03:06:10.479209 2371 leader_election.cc:302] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 278332e68f62492e90cbeb5802b32f8e, ffc2110a66234d14a58b23e4ff09ea0a; no voters:
I20260926 03:06:10.479562 2677 raft_consensus.cc:2781] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:10.479739 2677 raft_consensus.cc:686] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a [term 1 LEADER]: Becoming Leader. State: Replica: ffc2110a66234d14a58b23e4ff09ea0a, State: Running, Role: LEADER
I20260926 03:06:10.479997 2677 consensus_queue.cc:237] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:10.482895 2142 catalog_manager.cc:5495] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a reported cstate change: term changed from 0 to 1, leader changed from <none> to ffc2110a66234d14a58b23e4ff09ea0a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } health_report { overall_health: UNKNOWN } } }
I20260926 03:06:10.521801 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:10.515902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39744 (local address 127.0.97.2:39487)
0926 03:06:10.517057 (+ 1155us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:10.517064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:10.517086 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:06:10.517149 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:10.517153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:10.517217 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:10.517313 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:06:10.517721 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:10.518552 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:10.519951 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:10.520414 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:10.520584 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:10.521033 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:06:10.521148 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:06:10.521235 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:10.521402 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 03:06:10.521500 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":984,"thread_start_us":845,"threads_started":1}
I20260926 03:06:10.538151 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:10.532281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45809 (local address 127.0.97.2:39487)
0926 03:06:10.532465 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:10.532471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:10.532870 (+ 399us) server_negotiation.cc:407] Connection header received
0926 03:06:10.533019 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:10.533024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:10.533102 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:10.533198 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:06:10.533735 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:10.534790 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:10.536295 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:10.536889 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:10.537009 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:10.537431 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:06:10.537540 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:06:10.537616 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:10.537809 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 03:06:10.537874 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 03:06:10.538156 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:10.532213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39487 (local address 127.0.97.2:45809)
0926 03:06:10.532737 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:10.532752 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:10.532876 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:10.533227 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:10.533232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:10.533244 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:06:10.533547 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:10.533557 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:10.534932 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:10.534936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:10.536119 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:10.536132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:10.537106 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:10.537113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:10.537236 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:10.537276 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:06:10.537637 (+ 361us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:06:10.537646 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 03:06:10.537713 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 03:06:10.537824 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":303,"threads_started":1}
I20260926 03:06:10.546020 2579 raft_consensus.cc:1261] T 6f781701dae34974ac9c84b575565ab9 P 278332e68f62492e90cbeb5802b32f8e [term 1 FOLLOWER]: Refusing update from remote peer ffc2110a66234d14a58b23e4ff09ea0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:10.546118 2301 raft_consensus.cc:1261] T 6f781701dae34974ac9c84b575565ab9 P b15866fca8ae4ade8630d4f36ef468d4 [term 1 FOLLOWER]: Refusing update from remote peer ffc2110a66234d14a58b23e4ff09ea0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:10.546659 2719 consensus_queue.cc:1035] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:10.546934 2724 consensus_queue.cc:1035] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:10.598395 2141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:06:10.598680 2141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:06:10.602017 2141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b80281939834119bff32889ec34ea5b: Sending DeleteTablet for 3 replicas of tablet 6f781701dae34974ac9c84b575565ab9
I20260926 03:06:10.602921 2281 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f781701dae34974ac9c84b575565ab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60564
I20260926 03:06:10.603286 2738 tablet_replica.cc:330] T 6f781701dae34974ac9c84b575565ab9 P b15866fca8ae4ade8630d4f36ef468d4: stopping tablet replica
I20260926 03:06:10.603405 2559 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f781701dae34974ac9c84b575565ab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34764
I20260926 03:06:10.603508 2738 raft_consensus.cc:2229] T 6f781701dae34974ac9c84b575565ab9 P b15866fca8ae4ade8630d4f36ef468d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:06:10.603650 2740 tablet_replica.cc:330] T 6f781701dae34974ac9c84b575565ab9 P 278332e68f62492e90cbeb5802b32f8e: stopping tablet replica
I20260926 03:06:10.603699 2738 raft_consensus.cc:2258] T 6f781701dae34974ac9c84b575565ab9 P b15866fca8ae4ade8630d4f36ef468d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:10.603832 2740 raft_consensus.cc:2229] T 6f781701dae34974ac9c84b575565ab9 P 278332e68f62492e90cbeb5802b32f8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:06:10.604014 2740 raft_consensus.cc:2258] T 6f781701dae34974ac9c84b575565ab9 P 278332e68f62492e90cbeb5802b32f8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:10.604569 2420 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f781701dae34974ac9c84b575565ab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39730
I20260926 03:06:10.604878 2739 tablet_replica.cc:330] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a: stopping tablet replica
I20260926 03:06:10.605072 2739 raft_consensus.cc:2229] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:06:10.605165 2740 ts_tablet_manager.cc:1905] T 6f781701dae34974ac9c84b575565ab9 P 278332e68f62492e90cbeb5802b32f8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:10.605645 2739 raft_consensus.cc:2258] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:10.606472 2739 ts_tablet_manager.cc:1905] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:10.609326 2738 ts_tablet_manager.cc:1905] T 6f781701dae34974ac9c84b575565ab9 P b15866fca8ae4ade8630d4f36ef468d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:10.609869 2141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:06:10.610489 2141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:06:10.616438 2739 ts_tablet_manager.cc:1918] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:06:10.617522 2739 log.cc:1196] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-1/wal/wals/6f781701dae34974ac9c84b575565ab9
I20260926 03:06:10.618090 2739 ts_tablet_manager.cc:1939] T 6f781701dae34974ac9c84b575565ab9 P ffc2110a66234d14a58b23e4ff09ea0a: Deleting consensus metadata
I20260926 03:06:10.620147 2738 ts_tablet_manager.cc:1918] T 6f781701dae34974ac9c84b575565ab9 P b15866fca8ae4ade8630d4f36ef468d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:06:10.620431 2738 log.cc:1196] T 6f781701dae34974ac9c84b575565ab9 P b15866fca8ae4ade8630d4f36ef468d4: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-0/wal/wals/6f781701dae34974ac9c84b575565ab9
I20260926 03:06:10.620921 2738 ts_tablet_manager.cc:1939] T 6f781701dae34974ac9c84b575565ab9 P b15866fca8ae4ade8630d4f36ef468d4: Deleting consensus metadata
I20260926 03:06:10.623726 2281 tablet_service.cc:1467] Processing CreateTablet for tablet 843aa80dc4a74f8bad089f32280883a4 (DEFAULT_TABLE table=test-table [id=bf55c1d7e42b41daa4f6e6868e543f1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:10.623965 2559 tablet_service.cc:1467] Processing CreateTablet for tablet 843aa80dc4a74f8bad089f32280883a4 (DEFAULT_TABLE table=test-table [id=bf55c1d7e42b41daa4f6e6868e543f1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:10.624230 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843aa80dc4a74f8bad089f32280883a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:10.624210 2420 tablet_service.cc:1467] Processing CreateTablet for tablet 843aa80dc4a74f8bad089f32280883a4 (DEFAULT_TABLE table=test-table [id=bf55c1d7e42b41daa4f6e6868e543f1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:10.624606 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843aa80dc4a74f8bad089f32280883a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:10.629758 2750 tablet_bootstrap.cc:492] T 843aa80dc4a74f8bad089f32280883a4 P ffc2110a66234d14a58b23e4ff09ea0a: Bootstrap starting.
I20260926 03:06:10.631507 2751 tablet_bootstrap.cc:492] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4: Bootstrap starting.
I20260926 03:06:10.631758 2750 tablet_bootstrap.cc:654] T 843aa80dc4a74f8bad089f32280883a4 P ffc2110a66234d14a58b23e4ff09ea0a: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:10.633325 2751 tablet_bootstrap.cc:654] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:10.633756 2750 tablet_bootstrap.cc:492] T 843aa80dc4a74f8bad089f32280883a4 P ffc2110a66234d14a58b23e4ff09ea0a: No bootstrap required, opened a new log
I20260926 03:06:10.633924 2750 ts_tablet_manager.cc:1397] T 843aa80dc4a74f8bad089f32280883a4 P ffc2110a66234d14a58b23e4ff09ea0a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 03:06:10.634953 2750 raft_consensus.cc:348] T 843aa80dc4a74f8bad089f32280883a4 P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:10.635248 2750 raft_consensus.cc:374] T 843aa80dc4a74f8bad089f32280883a4 P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:10.635386 2750 raft_consensus.cc:729] T 843aa80dc4a74f8bad089f32280883a4 P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffc2110a66234d14a58b23e4ff09ea0a, State: Initialized, Role: FOLLOWER
I20260926 03:06:10.635701 2750 consensus_queue.cc:260] T 843aa80dc4a74f8bad089f32280883a4 P ffc2110a66234d14a58b23e4ff09ea0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:10.636456 2750 ts_tablet_manager.cc:1428] T 843aa80dc4a74f8bad089f32280883a4 P ffc2110a66234d14a58b23e4ff09ea0a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 03:06:10.640769 2751 tablet_bootstrap.cc:492] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4: No bootstrap required, opened a new log
I20260926 03:06:10.640954 2751 ts_tablet_manager.cc:1397] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260926 03:06:10.641628 2751 raft_consensus.cc:348] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:10.641844 2751 raft_consensus.cc:374] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:10.641916 2751 raft_consensus.cc:729] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b15866fca8ae4ade8630d4f36ef468d4, State: Initialized, Role: FOLLOWER
I20260926 03:06:10.642150 2751 consensus_queue.cc:260] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:10.642740 2751 ts_tablet_manager.cc:1428] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:06:10.671581 2128 catalog_manager.cc:4841] TS b15866fca8ae4ade8630d4f36ef468d4 (127.0.97.1:45529): tablet 6f781701dae34974ac9c84b575565ab9 (table test-table [id=4b683ca0e12c40fe95fd144a17563dba]) successfully deleted
I20260926 03:06:10.672554 2128 catalog_manager.cc:4841] TS ffc2110a66234d14a58b23e4ff09ea0a (127.0.97.2:39487): tablet 6f781701dae34974ac9c84b575565ab9 (table test-table [id=4b683ca0e12c40fe95fd144a17563dba]) successfully deleted
I20260926 03:06:10.691497 2559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843aa80dc4a74f8bad089f32280883a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:10.691497 2740 ts_tablet_manager.cc:1918] T 6f781701dae34974ac9c84b575565ab9 P 278332e68f62492e90cbeb5802b32f8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:06:10.691720 2740 log.cc:1196] T 6f781701dae34974ac9c84b575565ab9 P 278332e68f62492e90cbeb5802b32f8e: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-2/wal/wals/6f781701dae34974ac9c84b575565ab9
I20260926 03:06:10.692154 2740 ts_tablet_manager.cc:1939] T 6f781701dae34974ac9c84b575565ab9 P 278332e68f62492e90cbeb5802b32f8e: Deleting consensus metadata
I20260926 03:06:10.693943 2128 catalog_manager.cc:4841] TS 278332e68f62492e90cbeb5802b32f8e (127.0.97.3:34285): tablet 6f781701dae34974ac9c84b575565ab9 (table test-table [id=4b683ca0e12c40fe95fd144a17563dba]) successfully deleted
I20260926 03:06:10.697712 2749 tablet_bootstrap.cc:492] T 843aa80dc4a74f8bad089f32280883a4 P 278332e68f62492e90cbeb5802b32f8e: Bootstrap starting.
I20260926 03:06:10.699193 2749 tablet_bootstrap.cc:654] T 843aa80dc4a74f8bad089f32280883a4 P 278332e68f62492e90cbeb5802b32f8e: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:10.701308 2749 tablet_bootstrap.cc:492] T 843aa80dc4a74f8bad089f32280883a4 P 278332e68f62492e90cbeb5802b32f8e: No bootstrap required, opened a new log
I20260926 03:06:10.701500 2749 ts_tablet_manager.cc:1397] T 843aa80dc4a74f8bad089f32280883a4 P 278332e68f62492e90cbeb5802b32f8e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 03:06:10.702361 2749 raft_consensus.cc:348] T 843aa80dc4a74f8bad089f32280883a4 P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:10.702651 2749 raft_consensus.cc:374] T 843aa80dc4a74f8bad089f32280883a4 P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:10.702732 2749 raft_consensus.cc:729] T 843aa80dc4a74f8bad089f32280883a4 P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 278332e68f62492e90cbeb5802b32f8e, State: Initialized, Role: FOLLOWER
I20260926 03:06:10.702978 2749 consensus_queue.cc:260] T 843aa80dc4a74f8bad089f32280883a4 P 278332e68f62492e90cbeb5802b32f8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:10.703533 2749 ts_tablet_manager.cc:1428] T 843aa80dc4a74f8bad089f32280883a4 P 278332e68f62492e90cbeb5802b32f8e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:06:10.744328 2755 raft_consensus.cc:482] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:06:10.744544 2755 raft_consensus.cc:504] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:10.746129 2755 leader_election.cc:288] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffc2110a66234d14a58b23e4ff09ea0a (127.0.97.2:39487), 278332e68f62492e90cbeb5802b32f8e (127.0.97.3:34285)
I20260926 03:06:10.756588 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:10.746601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55887 (local address 127.0.97.2:39487)
0926 03:06:10.746794 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:10.746800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:10.747160 (+ 360us) server_negotiation.cc:407] Connection header received
0926 03:06:10.747445 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:10.747452 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:10.747536 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:10.747609 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:10.748456 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:10.749816 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:10.753163 (+ 3347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:10.755495 (+ 2332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:10.755644 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:10.755729 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:10.756138 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 03:06:10.756282 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:06:10.756628 2769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:10.746450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39487 (local address 127.0.97.1:55887)
0926 03:06:10.747035 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:10.747068 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:10.747221 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:10.747837 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:10.747843 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:10.747875 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:10.748261 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:10.748274 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:10.750099 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:10.750105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:10.752844 (+ 2739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:10.752872 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:10.755742 (+ 2870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:10.755749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:10.755959 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:10.756001 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 03:06:10.756101 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 03:06:10.756246 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":316,"threads_started":1}
I20260926 03:06:10.757601 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843aa80dc4a74f8bad089f32280883a4" candidate_uuid: "b15866fca8ae4ade8630d4f36ef468d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" is_pre_election: true
I20260926 03:06:10.757889 2440 raft_consensus.cc:2454] T 843aa80dc4a74f8bad089f32280883a4 P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b15866fca8ae4ade8630d4f36ef468d4 in term 0.
I20260926 03:06:10.758555 2231 leader_election.cc:302] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b15866fca8ae4ade8630d4f36ef468d4, ffc2110a66234d14a58b23e4ff09ea0a; no voters:
I20260926 03:06:10.759078 2755 raft_consensus.cc:2781] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:06:10.759227 2755 raft_consensus.cc:482] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:06:10.759331 2755 raft_consensus.cc:3037] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:10.763280 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:10.747425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57885 (local address 127.0.97.3:34285)
0926 03:06:10.755323 (+ 7898us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:10.755330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:10.755750 (+ 420us) server_negotiation.cc:407] Connection header received
0926 03:06:10.755986 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:10.755991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:10.756091 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:10.756190 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:10.757122 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:10.758193 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:10.760631 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:10.761681 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:10.762293 (+ 612us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:10.762424 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:10.762814 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 03:06:10.762956 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7730,"thread_start_us":7573,"threads_started":1}
I20260926 03:06:10.763288 2770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:10.747220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34285 (local address 127.0.97.1:57885)
0926 03:06:10.755619 (+ 8399us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:10.755639 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:10.755806 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:10.756374 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:10.756379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:10.756414 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:10.756817 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:10.756830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:10.758352 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:10.758358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:10.760419 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:10.760436 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:10.762440 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:10.762447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:10.762659 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:10.762685 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 03:06:10.762774 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 03:06:10.762920 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8239,"thread_start_us":8060,"threads_started":1}
I20260926 03:06:10.763991 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843aa80dc4a74f8bad089f32280883a4" candidate_uuid: "b15866fca8ae4ade8630d4f36ef468d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "278332e68f62492e90cbeb5802b32f8e" is_pre_election: true
I20260926 03:06:10.764281 2578 raft_consensus.cc:2454] T 843aa80dc4a74f8bad089f32280883a4 P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b15866fca8ae4ade8630d4f36ef468d4 in term 0.
I20260926 03:06:10.791842 2755 raft_consensus.cc:504] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:10.792830 2755 leader_election.cc:288] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4 [CANDIDATE]: Term 1 election: Requested vote from peers ffc2110a66234d14a58b23e4ff09ea0a (127.0.97.2:39487), 278332e68f62492e90cbeb5802b32f8e (127.0.97.3:34285)
I20260926 03:06:10.793350 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843aa80dc4a74f8bad089f32280883a4" candidate_uuid: "b15866fca8ae4ade8630d4f36ef468d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffc2110a66234d14a58b23e4ff09ea0a"
I20260926 03:06:10.793587 2440 raft_consensus.cc:3037] T 843aa80dc4a74f8bad089f32280883a4 P ffc2110a66234d14a58b23e4ff09ea0a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:10.794219 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843aa80dc4a74f8bad089f32280883a4" candidate_uuid: "b15866fca8ae4ade8630d4f36ef468d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "278332e68f62492e90cbeb5802b32f8e"
I20260926 03:06:10.794430 2578 raft_consensus.cc:3037] T 843aa80dc4a74f8bad089f32280883a4 P 278332e68f62492e90cbeb5802b32f8e [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:10.797410 2440 raft_consensus.cc:2454] T 843aa80dc4a74f8bad089f32280883a4 P ffc2110a66234d14a58b23e4ff09ea0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b15866fca8ae4ade8630d4f36ef468d4 in term 1.
I20260926 03:06:10.797978 2578 raft_consensus.cc:2454] T 843aa80dc4a74f8bad089f32280883a4 P 278332e68f62492e90cbeb5802b32f8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b15866fca8ae4ade8630d4f36ef468d4 in term 1.
I20260926 03:06:10.797995 2231 leader_election.cc:302] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b15866fca8ae4ade8630d4f36ef468d4, ffc2110a66234d14a58b23e4ff09ea0a; no voters:
I20260926 03:06:10.798319 2755 raft_consensus.cc:2781] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:10.798965 2755 raft_consensus.cc:686] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4 [term 1 LEADER]: Becoming Leader. State: Replica: b15866fca8ae4ade8630d4f36ef468d4, State: Running, Role: LEADER
I20260926 03:06:10.799417 2755 consensus_queue.cc:237] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } }
I20260926 03:06:10.803434 2142 catalog_manager.cc:5495] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b15866fca8ae4ade8630d4f36ef468d4 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "b15866fca8ae4ade8630d4f36ef468d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b15866fca8ae4ade8630d4f36ef468d4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 } health_report { overall_health: UNKNOWN } } }
I20260926 03:06:10.856367 2774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:10.845715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60568 (local address 127.0.97.1:45529)
0926 03:06:10.849328 (+ 3613us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:10.849335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:10.849365 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:06:10.849484 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:10.849489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:10.849611 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:10.849825 (+ 214us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:06:10.850730 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:10.851885 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:10.853389 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:10.854007 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:10.854159 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:10.854649 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:06:10.855531 (+ 882us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:06:10.855688 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:10.855879 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 03:06:10.855980 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3452,"thread_start_us":3274,"threads_started":1}
I20260926 03:06:10.873905 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:10.868334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59655 (local address 127.0.97.2:39487)
0926 03:06:10.868489 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:10.868493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:10.869086 (+ 593us) server_negotiation.cc:407] Connection header received
0926 03:06:10.869285 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:10.869289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:10.869352 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:10.869429 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:06:10.869925 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:10.870736 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:10.872107 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:10.872614 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:10.872739 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:10.873167 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:06:10.873296 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:06:10.873372 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:10.873511 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 03:06:10.873594 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 03:06:10.873989 2777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:10.868264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39487 (local address 127.0.97.1:59655)
0926 03:06:10.868969 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:10.868985 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:10.869145 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:10.869457 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:10.869461 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:10.869474 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:06:10.869779 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:10.869789 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:10.870872 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:10.870876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:10.871947 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:10.871961 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:10.872796 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:10.872803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:10.872928 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:10.872990 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:06:10.873395 (+ 405us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:06:10.873406 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 03:06:10.873476 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 03:06:10.873644 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":386,"threads_started":1}
I20260926 03:06:10.884202 2578 raft_consensus.cc:1261] T 843aa80dc4a74f8bad089f32280883a4 P 278332e68f62492e90cbeb5802b32f8e [term 1 FOLLOWER]: Refusing update from remote peer b15866fca8ae4ade8630d4f36ef468d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:10.884361 2440 raft_consensus.cc:1261] T 843aa80dc4a74f8bad089f32280883a4 P ffc2110a66234d14a58b23e4ff09ea0a [term 1 FOLLOWER]: Refusing update from remote peer b15866fca8ae4ade8630d4f36ef468d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:10.884981 2755 consensus_queue.cc:1035] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "278332e68f62492e90cbeb5802b32f8e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:10.885370 2772 consensus_queue.cc:1035] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffc2110a66234d14a58b23e4ff09ea0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:10.915948 2774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:10.909488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39729 (local address 127.0.97.1:45529)
0926 03:06:10.910349 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:10.910355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:10.910373 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:06:10.910438 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:10.910443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:10.910565 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:10.910660 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:06:10.911109 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:10.912169 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:10.914087 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:10.914703 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:10.914814 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:10.915214 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:06:10.915320 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:06:10.915392 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:10.915545 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 03:06:10.915625 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738}
I20260926 03:06:10.916006 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:10.908460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45529 (local address 127.0.97.2:39729)
0926 03:06:10.908686 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:10.908708 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:10.908825 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:10.910711 (+ 1886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:10.910716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:10.910729 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:06:10.910942 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:10.910955 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:10.912350 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:10.912355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:10.913889 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:10.913903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:10.914842 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:10.914848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:10.915016 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:10.915055 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:06:10.915419 (+ 364us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:06:10.915429 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 03:06:10.915518 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 03:06:10.915658 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
W20260926 03:06:10.917208 2721 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 03:06:12.530463 2142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:06:12.530737 2142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:06:12.534845 2142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b80281939834119bff32889ec34ea5b: Sending DeleteTablet for 3 replicas of tablet 843aa80dc4a74f8bad089f32280883a4
I20260926 03:06:12.535676 2281 tablet_service.cc:1514] Processing DeleteTablet for tablet 843aa80dc4a74f8bad089f32280883a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60564
I20260926 03:06:12.535885 2420 tablet_service.cc:1514] Processing DeleteTablet for tablet 843aa80dc4a74f8bad089f32280883a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39730
I20260926 03:06:12.535925 2559 tablet_service.cc:1514] Processing DeleteTablet for tablet 843aa80dc4a74f8bad089f32280883a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34764
I20260926 03:06:12.536397 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 2215
I20260926 03:06:12.551093 2804 tablet_replica.cc:330] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4: stopping tablet replica
I20260926 03:06:12.551177 2806 tablet_replica.cc:330] T 843aa80dc4a74f8bad089f32280883a4 P 278332e68f62492e90cbeb5802b32f8e: stopping tablet replica
I20260926 03:06:12.551345 2805 tablet_replica.cc:330] T 843aa80dc4a74f8bad089f32280883a4 P ffc2110a66234d14a58b23e4ff09ea0a: stopping tablet replica
I20260926 03:06:12.551358 2806 raft_consensus.cc:2229] T 843aa80dc4a74f8bad089f32280883a4 P 278332e68f62492e90cbeb5802b32f8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:06:12.551594 2806 raft_consensus.cc:2258] T 843aa80dc4a74f8bad089f32280883a4 P 278332e68f62492e90cbeb5802b32f8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:12.551615 2804 raft_consensus.cc:2229] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:06:12.551589 2805 raft_consensus.cc:2229] T 843aa80dc4a74f8bad089f32280883a4 P ffc2110a66234d14a58b23e4ff09ea0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:06:12.552397 2804 raft_consensus.cc:2258] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:12.552428 2805 raft_consensus.cc:2258] T 843aa80dc4a74f8bad089f32280883a4 P ffc2110a66234d14a58b23e4ff09ea0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:12.553277 2806 ts_tablet_manager.cc:1905] T 843aa80dc4a74f8bad089f32280883a4 P 278332e68f62492e90cbeb5802b32f8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:12.553318 2805 ts_tablet_manager.cc:1905] T 843aa80dc4a74f8bad089f32280883a4 P ffc2110a66234d14a58b23e4ff09ea0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:12.554077 2804 ts_tablet_manager.cc:1905] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:12.560113 2805 ts_tablet_manager.cc:1918] T 843aa80dc4a74f8bad089f32280883a4 P ffc2110a66234d14a58b23e4ff09ea0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:06:12.560113 2806 ts_tablet_manager.cc:1918] T 843aa80dc4a74f8bad089f32280883a4 P 278332e68f62492e90cbeb5802b32f8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:06:12.560266 2806 log.cc:1196] T 843aa80dc4a74f8bad089f32280883a4 P 278332e68f62492e90cbeb5802b32f8e: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-2/wal/wals/843aa80dc4a74f8bad089f32280883a4
I20260926 03:06:12.560274 2805 log.cc:1196] T 843aa80dc4a74f8bad089f32280883a4 P ffc2110a66234d14a58b23e4ff09ea0a: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-1/wal/wals/843aa80dc4a74f8bad089f32280883a4
I20260926 03:06:12.560724 2806 ts_tablet_manager.cc:1939] T 843aa80dc4a74f8bad089f32280883a4 P 278332e68f62492e90cbeb5802b32f8e: Deleting consensus metadata
I20260926 03:06:12.560997 2804 ts_tablet_manager.cc:1918] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:06:12.561116 2805 ts_tablet_manager.cc:1939] T 843aa80dc4a74f8bad089f32280883a4 P ffc2110a66234d14a58b23e4ff09ea0a: Deleting consensus metadata
I20260926 03:06:12.561144 2804 log.cc:1196] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391950058021-388-0/minicluster-data/ts-0/wal/wals/843aa80dc4a74f8bad089f32280883a4
I20260926 03:06:12.561484 2804 ts_tablet_manager.cc:1939] T 843aa80dc4a74f8bad089f32280883a4 P b15866fca8ae4ade8630d4f36ef468d4: Deleting consensus metadata
I20260926 03:06:12.561939 2128 catalog_manager.cc:4841] TS 278332e68f62492e90cbeb5802b32f8e (127.0.97.3:34285): tablet 843aa80dc4a74f8bad089f32280883a4 (table test-table [id=bf55c1d7e42b41daa4f6e6868e543f1c]) successfully deleted
I20260926 03:06:12.562206 2128 catalog_manager.cc:4841] TS ffc2110a66234d14a58b23e4ff09ea0a (127.0.97.2:39487): tablet 843aa80dc4a74f8bad089f32280883a4 (table test-table [id=bf55c1d7e42b41daa4f6e6868e543f1c]) successfully deleted
I20260926 03:06:12.562402 2128 catalog_manager.cc:4841] TS b15866fca8ae4ade8630d4f36ef468d4 (127.0.97.1:45529): tablet 843aa80dc4a74f8bad089f32280883a4 (table test-table [id=bf55c1d7e42b41daa4f6e6868e543f1c]) successfully deleted
Sep 26 03:06:12 dist-test-slave-hmbw krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391970, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:06:12.577250 2808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:12.538018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60584 (local address 127.0.97.1:45529)
0926 03:06:12.552577 (+ 14559us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:12.552583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:12.552613 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:06:12.552679 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:12.552684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:12.552768 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:12.552916 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:12.555480 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:12.556718 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:12.558676 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:12.559374 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:12.559505 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:12.570951 (+ 11446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:12.570968 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:12.570985 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:12.571027 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:12.574430 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:12.575214 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:12.575221 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:12.575223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:12.575352 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:12.575821 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:12.575826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:12.575828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:12.576212 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:12.576456 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:12.576733 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 03:06:12.576904 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14400,"thread_start_us":14472,"threads_started":1}
W20260926 03:06:12.577956 2346 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60584
W20260926 03:06:12.578639 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 2215: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:06:12.578886 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 2215
I20260926 03:06:12.597815 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 2355
Sep 26 03:06:12 dist-test-slave-hmbw krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391970, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:06:12.613083 2809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:12.598633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39748 (local address 127.0.97.2:39487)
0926 03:06:12.599135 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:12.599139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:12.599154 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:06:12.599202 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:12.599230 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:12.599287 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:12.599379 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:12.600579 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:12.601331 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:12.602675 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:12.603085 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:12.603191 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:12.607259 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:12.607275 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:12.607291 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:12.607342 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:12.610230 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:12.611078 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:12.611085 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:12.611088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:12.611234 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:12.611594 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:12.611598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:12.611600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:12.611958 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:12.612248 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:12.612540 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 03:06:12.612767 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":226,"threads_started":1}
W20260926 03:06:12.613802 2485 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39748
W20260926 03:06:12.614455 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 2355: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:06:12.614818 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 2355
I20260926 03:06:12.635622 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 2494
Sep 26 03:06:12 dist-test-slave-hmbw krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391970, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:06:12.651583 2810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:12.636431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34778 (local address 127.0.97.3:34285)
0926 03:06:12.636916 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:12.636920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:12.636938 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:06:12.636990 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:12.636994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:12.637055 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:12.637160 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:12.638556 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:12.639330 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:12.640553 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:12.640975 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:12.641078 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:12.645513 (+ 4435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:12.645529 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:12.645546 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:12.645592 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:12.648623 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:12.649409 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:12.649417 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:12.649419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:12.649605 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:12.650038 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:12.650041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:12.650043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:12.650415 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:12.650744 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:12.651092 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 03:06:12.651278 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":272,"threads_started":1}
W20260926 03:06:12.652258 2624 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34778
W20260926 03:06:12.652848 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 2494: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:06:12.653003 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 2494
I20260926 03:06:12.670413 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 2112
I20260926 03:06:12.671662 2184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:12.786099 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 2112
2026-09-26T03:06:12Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8770 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:06:12 dist-test-slave-hmbw krb5kdc[2814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:06:12 dist-test-slave-hmbw krb5kdc[2814](info): set up 2 sockets
Sep 26 03:06:12 dist-test-slave-hmbw krb5kdc[2814](info): commencing operation
krb5kdc: starting...
W20260926 03:06:13.794596 388 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:06:13 dist-test-slave-hmbw krb5kdc[2814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:06:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:06:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:06:13.987118 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42451
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:14.306931 2827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:14.307252 2827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:14.307375 2827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:14.314781 2827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:06:14.314913 2827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:14.314997 2827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:14.315075 2827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:06:14.315130 2827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:06:14.325356 2827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:14.326723 2827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:14.328027 2827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:14.339730 2832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:14.339658 2833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:14.339690 2835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:14.340557 2827 server_base.cc:1034] running on GCE node
I20260926 03:06:14.341368 2827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:14.342864 2827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:14.344136 2827 hybrid_clock.cc:648] HybridClock initialized: now 1790391974344085 us; error 42 us; skew 500 ppm
I20260926 03:06:14.344487 2827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:14 dist-test-slave-hmbw krb5kdc[2814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:14.347774 2827 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:06:14.349275 2827 webserver.cc:460] Webserver started at http://127.0.97.62:44371/ using document root <none> and password file <none>
I20260926 03:06:14.349967 2827 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:14.350070 2827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:14.350330 2827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:14.353272 2827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "f70d9ec7fda2451887ab569555157c49"
format_stamp: "Formatted at 2026-09-26 03:06:14 on dist-test-slave-hmbw"
server_key: "8360f765284361bfe9efdde40cb7c44f"
server_key_iv: "714eb8b8e9fd0e8163810e2a3449519d"
server_key_version: "encryptionkey@0"
I20260926 03:06:14.354061 2827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "f70d9ec7fda2451887ab569555157c49"
format_stamp: "Formatted at 2026-09-26 03:06:14 on dist-test-slave-hmbw"
server_key: "8360f765284361bfe9efdde40cb7c44f"
server_key_iv: "714eb8b8e9fd0e8163810e2a3449519d"
server_key_version: "encryptionkey@0"
I20260926 03:06:14.359340 2827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 03:06:14.362854 2842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:14.364454 2827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:06:14.364641 2827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "f70d9ec7fda2451887ab569555157c49"
format_stamp: "Formatted at 2026-09-26 03:06:14 on dist-test-slave-hmbw"
server_key: "8360f765284361bfe9efdde40cb7c44f"
server_key_iv: "714eb8b8e9fd0e8163810e2a3449519d"
server_key_version: "encryptionkey@0"
I20260926 03:06:14.364836 2827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:14.395607 2827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:14.399878 2827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:14.400107 2827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:14.417754 2827 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42451
I20260926 03:06:14.417863 2893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42451 every 8 connection(s)
I20260926 03:06:14.419117 2827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:06:14.422941 2894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:14.425231 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 2827
I20260926 03:06:14.425454 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:06:14.426223 388 external_mini_cluster.cc:1428] Setting key a94add4f02694b95c3c5f7ce269dee65
I20260926 03:06:14.435632 2894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f70d9ec7fda2451887ab569555157c49: Bootstrap starting.
I20260926 03:06:14.439502 2894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f70d9ec7fda2451887ab569555157c49: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:14.440979 2894 log.cc:824] T 00000000000000000000000000000000 P f70d9ec7fda2451887ab569555157c49: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:06:14 dist-test-slave-hmbw krb5kdc[2814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:14.444080 2894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f70d9ec7fda2451887ab569555157c49: No bootstrap required, opened a new log
I20260926 03:06:14.450080 2897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:14.428716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36518 (local address 127.0.97.62:42451)
0926 03:06:14.429760 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:14.429772 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:14.429814 (+ 42us) server_negotiation.cc:407] Connection header received
0926 03:06:14.431513 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:14.431537 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:14.431966 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:14.432313 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:14.434083 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:14.435526 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:14.437307 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:14.438034 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:14.438177 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:14.443087 (+ 4910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:14.443096 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:14.443111 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:14.443147 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:14.446041 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:14.446809 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:14.446816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:14.446824 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:14.446993 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:14.447357 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:14.447360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:14.447362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:14.447912 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:14.448209 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:14.448613 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 03:06:14.449040 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":548,"threads_started":1}
I20260926 03:06:14.451820 2894 raft_consensus.cc:348] T 00000000000000000000000000000000 P f70d9ec7fda2451887ab569555157c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70d9ec7fda2451887ab569555157c49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42451 } }
I20260926 03:06:14.452167 2894 raft_consensus.cc:374] T 00000000000000000000000000000000 P f70d9ec7fda2451887ab569555157c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:14.452260 2894 raft_consensus.cc:729] T 00000000000000000000000000000000 P f70d9ec7fda2451887ab569555157c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f70d9ec7fda2451887ab569555157c49, State: Initialized, Role: FOLLOWER
I20260926 03:06:14.453019 2894 consensus_queue.cc:260] T 00000000000000000000000000000000 P f70d9ec7fda2451887ab569555157c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70d9ec7fda2451887ab569555157c49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42451 } }
I20260926 03:06:14.453248 2894 raft_consensus.cc:388] T 00000000000000000000000000000000 P f70d9ec7fda2451887ab569555157c49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:06:14.453369 2894 raft_consensus.cc:482] T 00000000000000000000000000000000 P f70d9ec7fda2451887ab569555157c49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:06:14.453511 2894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f70d9ec7fda2451887ab569555157c49 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:14.455785 2894 raft_consensus.cc:504] T 00000000000000000000000000000000 P f70d9ec7fda2451887ab569555157c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70d9ec7fda2451887ab569555157c49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42451 } }
I20260926 03:06:14.456337 2894 leader_election.cc:302] T 00000000000000000000000000000000 P f70d9ec7fda2451887ab569555157c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f70d9ec7fda2451887ab569555157c49; no voters:
I20260926 03:06:14.456871 2894 leader_election.cc:288] T 00000000000000000000000000000000 P f70d9ec7fda2451887ab569555157c49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:06:14.456988 2899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f70d9ec7fda2451887ab569555157c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:14.457578 2899 raft_consensus.cc:686] T 00000000000000000000000000000000 P f70d9ec7fda2451887ab569555157c49 [term 1 LEADER]: Becoming Leader. State: Replica: f70d9ec7fda2451887ab569555157c49, State: Running, Role: LEADER
I20260926 03:06:14.458078 2899 consensus_queue.cc:237] T 00000000000000000000000000000000 P f70d9ec7fda2451887ab569555157c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70d9ec7fda2451887ab569555157c49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42451 } }
I20260926 03:06:14.458774 2894 sys_catalog.cc:564] T 00000000000000000000000000000000 P f70d9ec7fda2451887ab569555157c49 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:06:14.460821 2901 sys_catalog.cc:455] T 00000000000000000000000000000000 P f70d9ec7fda2451887ab569555157c49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f70d9ec7fda2451887ab569555157c49. Latest consensus state: current_term: 1 leader_uuid: "f70d9ec7fda2451887ab569555157c49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70d9ec7fda2451887ab569555157c49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42451 } } }
I20260926 03:06:14.460896 2900 sys_catalog.cc:455] T 00000000000000000000000000000000 P f70d9ec7fda2451887ab569555157c49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f70d9ec7fda2451887ab569555157c49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70d9ec7fda2451887ab569555157c49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42451 } } }
I20260926 03:06:14.461221 2901 sys_catalog.cc:458] T 00000000000000000000000000000000 P f70d9ec7fda2451887ab569555157c49 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:14.461254 2900 sys_catalog.cc:458] T 00000000000000000000000000000000 P f70d9ec7fda2451887ab569555157c49 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:14.462338 2907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:06:14.468472 2907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:06:14.478192 2907 catalog_manager.cc:1347] Generated new cluster ID: 5c5636fa2f674a9680373c05957f5192
I20260926 03:06:14.478319 2907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:06:14.489468 2907 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:06:14.491446 2907 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:06:14.504484 2907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f70d9ec7fda2451887ab569555157c49: Generated new TSK 0
I20260926 03:06:14.505695 2907 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:06:14.550402 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42451
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:14.872357 2920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:14.872660 2920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:14.872788 2920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:14.880192 2920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:14.880352 2920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:14.880504 2920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:06:14.890352 2920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:14.891635 2920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:14.892874 2920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:14.905850 2925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:14.906005 2926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:14.905978 2928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:14.906783 2920 server_base.cc:1034] running on GCE node
I20260926 03:06:14.907420 2920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:14.908382 2920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:14.909587 2920 hybrid_clock.cc:648] HybridClock initialized: now 1790391974909538 us; error 41 us; skew 500 ppm
I20260926 03:06:14.909889 2920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:14 dist-test-slave-hmbw krb5kdc[2814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:14.912926 2920 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:06:14.914608 2920 webserver.cc:460] Webserver started at http://127.0.97.1:34377/ using document root <none> and password file <none>
I20260926 03:06:14.915349 2920 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:14.915462 2920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:14.915766 2920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:14.920187 2920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/data/instance:
uuid: "5515acbfe3114b8a8b44a23198c79589"
format_stamp: "Formatted at 2026-09-26 03:06:14 on dist-test-slave-hmbw"
server_key: "217ed1c0b6b111eaf1e1756b7cc6e8a2"
server_key_iv: "03dbfe95fda197ea7147b8b8fef57c5d"
server_key_version: "encryptionkey@0"
I20260926 03:06:14.921388 2920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5515acbfe3114b8a8b44a23198c79589"
format_stamp: "Formatted at 2026-09-26 03:06:14 on dist-test-slave-hmbw"
server_key: "217ed1c0b6b111eaf1e1756b7cc6e8a2"
server_key_iv: "03dbfe95fda197ea7147b8b8fef57c5d"
server_key_version: "encryptionkey@0"
I20260926 03:06:14.928599 2920 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 03:06:14.932928 2935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:14.934692 2920 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:06:14.934882 2920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "5515acbfe3114b8a8b44a23198c79589"
format_stamp: "Formatted at 2026-09-26 03:06:14 on dist-test-slave-hmbw"
server_key: "217ed1c0b6b111eaf1e1756b7cc6e8a2"
server_key_iv: "03dbfe95fda197ea7147b8b8fef57c5d"
server_key_version: "encryptionkey@0"
I20260926 03:06:14.935104 2920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:14.951920 2920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:14.957036 2920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:14.957342 2920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:14.958490 2920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:06:14.960850 2920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:14.960978 2920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:14.961123 2920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:14.961269 2920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:15.003520 2920 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33071
I20260926 03:06:15.003530 3047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33071 every 8 connection(s)
I20260926 03:06:15.006480 2920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:06:15.010740 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 2920
I20260926 03:06:15.010960 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:06:15.011700 388 external_mini_cluster.cc:1428] Setting key 0b54fbea9c9b3bc0dbcb5f4156ecc288
Sep 26 03:06:15 dist-test-slave-hmbw krb5kdc[2814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:06:15.031975 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:15.009715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56557 (local address 127.0.97.62:42451)
0926 03:06:15.010237 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:15.010241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:15.011340 (+ 1099us) server_negotiation.cc:407] Connection header received
0926 03:06:15.013616 (+ 2276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:15.013622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:15.013702 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:15.013789 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:15.015563 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:15.016572 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:15.018733 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:15.019188 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:15.019326 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:15.024820 (+ 5494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:15.024828 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:15.024832 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:15.024890 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:15.028295 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:15.029263 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:15.029268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:15.029271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:15.029406 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:15.029842 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:15.029847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:15.029849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:15.030189 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:15.030347 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:15.031479 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0926 03:06:15.031674 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":273,"threads_started":1}
I20260926 03:06:15.032761 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:15.010216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42451 (local address 127.0.97.1:56557)
0926 03:06:15.011153 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:15.011201 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:15.013258 (+ 2057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:15.014080 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:15.014089 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:15.014702 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:15.015345 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:15.015365 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:15.016799 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:15.016804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:15.018344 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:15.018432 (+ 88us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:15.019557 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:15.019566 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:15.020049 (+ 483us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:15.021022 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:15.021059 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:15.024216 (+ 3157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:15.028527 (+ 4311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:15.028538 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:15.028555 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:15.029087 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:15.029516 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:15.029521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:15.029525 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:15.029717 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:15.030352 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:15.030360 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:15.030788 (+ 428us) client_negotiation.cc:769] Sending connection context
0926 03:06:15.031066 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 03:06:15.031547 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":345,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 03:06:15.035142 3048 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42451
I20260926 03:06:15.035504 3048 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:15.036300 3048 heartbeater.cc:511] Master 127.0.97.62:42451 requested a full tablet report, sending...
I20260926 03:06:15.039984 2859 ts_manager.cc:194] Registered new tserver with Master: 5515acbfe3114b8a8b44a23198c79589 (127.0.97.1:33071)
I20260926 03:06:15.041785 2859 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:06:15.069528 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42451
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:15.439464 3054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:15.439767 3054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:15.439860 3054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:15.447307 3054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:15.447444 3054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:15.447563 3054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:06:15.457695 3054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:15.459161 3054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:15.460600 3054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:15.474313 3059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:15.474254 3062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:15.474254 3060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:15.475169 3054 server_base.cc:1034] running on GCE node
I20260926 03:06:15.475772 3054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:15.476761 3054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:15.477972 3054 hybrid_clock.cc:648] HybridClock initialized: now 1790391975477916 us; error 43 us; skew 500 ppm
I20260926 03:06:15.478333 3054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:15 dist-test-slave-hmbw krb5kdc[2814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:15.481496 3054 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:06:15.483199 3054 webserver.cc:460] Webserver started at http://127.0.97.2:32877/ using document root <none> and password file <none>
I20260926 03:06:15.483919 3054 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:15.484041 3054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:15.484351 3054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:15.487867 3054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/data/instance:
uuid: "89b3080150e8444980ad6d5db2f3bae7"
format_stamp: "Formatted at 2026-09-26 03:06:15 on dist-test-slave-hmbw"
server_key: "8a150fc69e26f6208e7bfbf3b330da0d"
server_key_iv: "729241f3486c4686f71aa99b7bc4f9ba"
server_key_version: "encryptionkey@0"
I20260926 03:06:15.488703 3054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/wal/instance:
uuid: "89b3080150e8444980ad6d5db2f3bae7"
format_stamp: "Formatted at 2026-09-26 03:06:15 on dist-test-slave-hmbw"
server_key: "8a150fc69e26f6208e7bfbf3b330da0d"
server_key_iv: "729241f3486c4686f71aa99b7bc4f9ba"
server_key_version: "encryptionkey@0"
I20260926 03:06:15.494298 3054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 03:06:15.498426 3069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:15.500070 3054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:06:15.500262 3054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "89b3080150e8444980ad6d5db2f3bae7"
format_stamp: "Formatted at 2026-09-26 03:06:15 on dist-test-slave-hmbw"
server_key: "8a150fc69e26f6208e7bfbf3b330da0d"
server_key_iv: "729241f3486c4686f71aa99b7bc4f9ba"
server_key_version: "encryptionkey@0"
I20260926 03:06:15.500487 3054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:15.521538 3054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:15.526146 3054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:15.526448 3054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:15.527470 3054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:06:15.528779 3054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:15.528852 3054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:15.528973 3054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:15.529021 3054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:15.560263 3054 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35583
I20260926 03:06:15.560307 3181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35583 every 8 connection(s)
I20260926 03:06:15.561545 3054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:06:15.571167 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 3054
I20260926 03:06:15.571370 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:06:15.572036 388 external_mini_cluster.cc:1428] Setting key a03f25ecb40cdc0aa451d1d9991af027
Sep 26 03:06:15 dist-test-slave-hmbw krb5kdc[2814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:15.584663 3184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:15.564546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53943 (local address 127.0.97.62:42451)
0926 03:06:15.565053 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:15.565061 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:15.565977 (+ 916us) server_negotiation.cc:407] Connection header received
0926 03:06:15.568031 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:15.568035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:15.568126 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:15.568222 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:15.569930 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:15.570714 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:15.573182 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:15.573811 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:15.573937 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:15.578562 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:15.578573 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:15.578578 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:15.578620 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:15.581157 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:15.582068 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:15.582074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:15.582078 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:15.582240 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:15.582774 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:15.582779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:15.582782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:15.583129 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:15.583267 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:15.584079 (+ 812us) server_negotiation.cc:299] Negotiation successful
0926 03:06:15.584311 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":252,"threads_started":1}
I20260926 03:06:15.585592 3185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:15.564984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42451 (local address 127.0.97.2:53943)
0926 03:06:15.565786 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:15.565832 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:15.567693 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:15.568499 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:15.568508 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:15.569075 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:15.569724 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:15.569744 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:15.570891 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:15.570897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:15.573004 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:15.573022 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:15.574066 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:15.574080 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:15.574373 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:15.575158 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:15.575191 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:15.578139 (+ 2948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:15.581355 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:15.581364 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:15.581380 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:15.581897 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:15.582389 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:15.582393 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:15.582396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:15.582657 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:15.583265 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:15.583272 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:15.583648 (+ 376us) client_negotiation.cc:769] Sending connection context
0926 03:06:15.583958 (+ 310us) client_negotiation.cc:240] Negotiation successful
0926 03:06:15.584421 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":320,"threads_started":1}
I20260926 03:06:15.587126 3182 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42451
I20260926 03:06:15.587469 3182 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:15.588395 3182 heartbeater.cc:511] Master 127.0.97.62:42451 requested a full tablet report, sending...
I20260926 03:06:15.590119 2859 ts_manager.cc:194] Registered new tserver with Master: 89b3080150e8444980ad6d5db2f3bae7 (127.0.97.2:35583)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 03:06:15.591084 2859 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:06:15.619647 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42451
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:15.959731 3188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:15.960033 3188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:15.960155 3188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:15.967605 3188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:15.967721 3188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:15.967861 3188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:06:15.978286 3188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:15.979614 3188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:15.980898 3188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:15.993650 3193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:15.993621 3196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:15.993687 3194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:15.994467 3188 server_base.cc:1034] running on GCE node
I20260926 03:06:15.995172 3188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:15.996286 3188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:15.997521 3188 hybrid_clock.cc:648] HybridClock initialized: now 1790391975997455 us; error 69 us; skew 500 ppm
I20260926 03:06:15.997852 3188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:15 dist-test-slave-hmbw krb5kdc[2814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:16.001027 3188 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:06:16.002766 3188 webserver.cc:460] Webserver started at http://127.0.97.3:42121/ using document root <none> and password file <none>
I20260926 03:06:16.003453 3188 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:16.003575 3188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:16.003902 3188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:16.007062 3188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/data/instance:
uuid: "d6a17d9ba61a48e0ad2edea3f91b65aa"
format_stamp: "Formatted at 2026-09-26 03:06:16 on dist-test-slave-hmbw"
server_key: "82d86bac757d4affb97b5db905b64193"
server_key_iv: "d5d347e797f68d67a86fb55dbee5fb70"
server_key_version: "encryptionkey@0"
I20260926 03:06:16.007845 3188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d6a17d9ba61a48e0ad2edea3f91b65aa"
format_stamp: "Formatted at 2026-09-26 03:06:16 on dist-test-slave-hmbw"
server_key: "82d86bac757d4affb97b5db905b64193"
server_key_iv: "d5d347e797f68d67a86fb55dbee5fb70"
server_key_version: "encryptionkey@0"
I20260926 03:06:16.013345 3188 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 03:06:16.017446 3203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:16.019112 3188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 03:06:16.019275 3188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "d6a17d9ba61a48e0ad2edea3f91b65aa"
format_stamp: "Formatted at 2026-09-26 03:06:16 on dist-test-slave-hmbw"
server_key: "82d86bac757d4affb97b5db905b64193"
server_key_iv: "d5d347e797f68d67a86fb55dbee5fb70"
server_key_version: "encryptionkey@0"
I20260926 03:06:16.019470 3188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:16.044374 3188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:16.046288 3048 heartbeater.cc:503] Master 127.0.97.62:42451 was elected leader, sending a full tablet report...
I20260926 03:06:16.049289 3188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:16.049535 3188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:16.050663 3188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:06:16.052117 3188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:16.052194 3188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:16.052338 3188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:16.052380 3188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:16.089534 3188 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41731
I20260926 03:06:16.089573 3315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41731 every 8 connection(s)
I20260926 03:06:16.091296 3188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:06:16.098693 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 3188
I20260926 03:06:16.098865 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:06:16.099253 388 external_mini_cluster.cc:1428] Setting key a8f241865f5760d5935177932f9c6bb9
Sep 26 03:06:16 dist-test-slave-hmbw krb5kdc[2814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:16.111920 3318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:16.093931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58963 (local address 127.0.97.62:42451)
0926 03:06:16.094463 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:16.094469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:16.095311 (+ 842us) server_negotiation.cc:407] Connection header received
0926 03:06:16.097402 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:16.097407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:16.097485 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:16.097591 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:16.099165 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:16.099997 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:16.101594 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:16.102013 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:16.102120 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:16.105754 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:16.105759 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:16.105762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:16.105798 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:16.108589 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:16.109572 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:16.109576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:16.109579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:16.109697 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:16.110135 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:16.110139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:16.110141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:16.110400 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:16.110558 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:16.111294 (+ 736us) server_negotiation.cc:299] Negotiation successful
0926 03:06:16.111540 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":281,"threads_started":1}
I20260926 03:06:16.112893 3319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:16.094416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42451 (local address 127.0.97.3:58963)
0926 03:06:16.095152 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:16.095195 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:16.097103 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:16.097782 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:16.097790 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:16.098300 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:16.098986 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:16.099007 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:16.100161 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:16.100167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:16.101441 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:16.101454 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:16.102227 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:16.102234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:16.102456 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:16.103147 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:16.103170 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:16.105478 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:16.108802 (+ 3324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:16.108811 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:16.108826 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:16.109441 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:16.109806 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:16.109811 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:16.109815 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:16.110020 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:16.110552 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:16.110561 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:16.110908 (+ 347us) client_negotiation.cc:769] Sending connection context
0926 03:06:16.111150 (+ 242us) client_negotiation.cc:240] Negotiation successful
0926 03:06:16.111733 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":246,"threads_started":1}
I20260926 03:06:16.114291 3316 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42451
I20260926 03:06:16.114645 3316 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:16.115440 3316 heartbeater.cc:511] Master 127.0.97.62:42451 requested a full tablet report, sending...
I20260926 03:06:16.117072 2859 ts_manager.cc:194] Registered new tserver with Master: d6a17d9ba61a48e0ad2edea3f91b65aa (127.0.97.3:41731)
I20260926 03:06:16.117890 2859 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58963
I20260926 03:06:16.127710 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 03:06:16.158463 3318 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:06:16.154917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56648 (local address 127.0.97.62:42451)
0926 03:06:16.155223 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:16.155229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:16.156079 (+ 850us) server_negotiation.cc:407] Connection header received
0926 03:06:16.156248 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:16.156252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:16.156322 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:16.156415 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:16.158185 (+ 1770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56648: BlockingRecv error: recv got EOF from 127.0.0.1:56648 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 03:06:16.162521 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 2920
W20260926 03:06:16.165568 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 2920: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:33071: server requires authentication, but client does not have Kerberos credentials available
I20260926 03:06:16.165745 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 2920
I20260926 03:06:16.178927 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 3054
W20260926 03:06:16.181578 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 3054: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:35583: server requires authentication, but client does not have Kerberos credentials available
I20260926 03:06:16.181749 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 3054
I20260926 03:06:16.194335 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 3188
W20260926 03:06:16.197870 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 3188: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:41731: server requires authentication, but client does not have Kerberos credentials available
I20260926 03:06:16.198019 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 3188
I20260926 03:06:16.210968 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 2827
I20260926 03:06:16.212189 2889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:16.314740 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 2827
2026-09-26T03:06:16Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3526 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:06:16 dist-test-slave-hmbw krb5kdc[3334](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:06:16 dist-test-slave-hmbw krb5kdc[3334](info): set up 2 sockets
Sep 26 03:06:16 dist-test-slave-hmbw krb5kdc[3334](info): commencing operation
krb5kdc: starting...
W20260926 03:06:17.390688 388 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:06:17 dist-test-slave-hmbw krb5kdc[3334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:06:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:06:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:06:17.583514 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35871
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:17.905139 3347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:17.905452 3347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:17.905571 3347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:17.912801 3347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:06:17.912935 3347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:17.913012 3347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:17.913085 3347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:06:17.913151 3347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:06:17.923203 3347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:17.924544 3347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:17.925917 3347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:17.937979 3353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:17.937897 3352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:17.938007 3355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:17.938808 3347 server_base.cc:1034] running on GCE node
I20260926 03:06:17.939579 3347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:17.940948 3347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:17.942145 3347 hybrid_clock.cc:648] HybridClock initialized: now 1790391977942101 us; error 30 us; skew 500 ppm
I20260926 03:06:17.942449 3347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:17 dist-test-slave-hmbw krb5kdc[3334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:17.950320 3347 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:06:17.951915 3347 webserver.cc:460] Webserver started at http://127.0.97.62:43761/ using document root <none> and password file <none>
I20260926 03:06:17.952612 3347 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:17.952715 3347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:17.952967 3347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:17.956050 3347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "7da4634985964688b08c580e06eb6c63"
format_stamp: "Formatted at 2026-09-26 03:06:17 on dist-test-slave-hmbw"
server_key: "6e48dc98d62aa0120e5267fb25058242"
server_key_iv: "ba0fdbeabcace591e6fa17e83f892a4b"
server_key_version: "encryptionkey@0"
I20260926 03:06:17.956846 3347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "7da4634985964688b08c580e06eb6c63"
format_stamp: "Formatted at 2026-09-26 03:06:17 on dist-test-slave-hmbw"
server_key: "6e48dc98d62aa0120e5267fb25058242"
server_key_iv: "ba0fdbeabcace591e6fa17e83f892a4b"
server_key_version: "encryptionkey@0"
I20260926 03:06:17.963317 3347 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 03:06:17.967051 3362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:17.968451 3347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:06:17.968621 3347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "7da4634985964688b08c580e06eb6c63"
format_stamp: "Formatted at 2026-09-26 03:06:17 on dist-test-slave-hmbw"
server_key: "6e48dc98d62aa0120e5267fb25058242"
server_key_iv: "ba0fdbeabcace591e6fa17e83f892a4b"
server_key_version: "encryptionkey@0"
I20260926 03:06:17.968811 3347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:17.991436 3347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:17.995944 3347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:17.996321 3347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:18.015376 3347 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35871
I20260926 03:06:18.015406 3413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35871 every 8 connection(s)
I20260926 03:06:18.016762 3347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:06:18.020685 3414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:18.021389 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 3347
I20260926 03:06:18.021593 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:06:18.022368 388 external_mini_cluster.cc:1428] Setting key 4462f6b2fc008a3824784dd10f2fa868
I20260926 03:06:18.035648 3414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7da4634985964688b08c580e06eb6c63: Bootstrap starting.
Sep 26 03:06:18 dist-test-slave-hmbw krb5kdc[3334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:18.039667 3414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7da4634985964688b08c580e06eb6c63: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:18.040884 3414 log.cc:824] T 00000000000000000000000000000000 P 7da4634985964688b08c580e06eb6c63: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:18.044180 3414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7da4634985964688b08c580e06eb6c63: No bootstrap required, opened a new log
I20260926 03:06:18.047637 3417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:18.024601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53768 (local address 127.0.97.62:35871)
0926 03:06:18.025529 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:18.025542 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:18.025590 (+ 48us) server_negotiation.cc:407] Connection header received
0926 03:06:18.027341 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:18.027367 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:18.027835 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:18.028213 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:18.030318 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:18.031725 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:18.033528 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:18.034255 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:18.034404 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:18.039366 (+ 4962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:18.039375 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:18.039392 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:18.039446 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:18.042776 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:18.043675 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:18.043685 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:18.043694 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:18.043938 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:18.044373 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:18.044378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:18.044382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:18.044996 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:18.045227 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:18.045787 (+ 560us) server_negotiation.cc:299] Negotiation successful
0926 03:06:18.046222 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":410,"threads_started":1}
I20260926 03:06:18.052030 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:06:18.053264 3414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7da4634985964688b08c580e06eb6c63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da4634985964688b08c580e06eb6c63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35871 } }
I20260926 03:06:18.053651 3414 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7da4634985964688b08c580e06eb6c63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:18.053747 3414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7da4634985964688b08c580e06eb6c63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da4634985964688b08c580e06eb6c63, State: Initialized, Role: FOLLOWER
I20260926 03:06:18.054421 3414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7da4634985964688b08c580e06eb6c63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da4634985964688b08c580e06eb6c63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35871 } }
I20260926 03:06:18.054690 3414 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7da4634985964688b08c580e06eb6c63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:06:18.054819 3414 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7da4634985964688b08c580e06eb6c63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:06:18.054971 3414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7da4634985964688b08c580e06eb6c63 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:18.057408 3414 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7da4634985964688b08c580e06eb6c63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da4634985964688b08c580e06eb6c63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35871 } }
I20260926 03:06:18.058012 3414 leader_election.cc:302] T 00000000000000000000000000000000 P 7da4634985964688b08c580e06eb6c63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7da4634985964688b08c580e06eb6c63; no voters:
I20260926 03:06:18.058660 3414 leader_election.cc:288] T 00000000000000000000000000000000 P 7da4634985964688b08c580e06eb6c63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:06:18.058789 3420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7da4634985964688b08c580e06eb6c63 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:18.061120 3414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7da4634985964688b08c580e06eb6c63 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:06:18.065070 3420 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7da4634985964688b08c580e06eb6c63 [term 1 LEADER]: Becoming Leader. State: Replica: 7da4634985964688b08c580e06eb6c63, State: Running, Role: LEADER
I20260926 03:06:18.065841 3420 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7da4634985964688b08c580e06eb6c63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da4634985964688b08c580e06eb6c63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35871 } }
I20260926 03:06:18.069029 3421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7da4634985964688b08c580e06eb6c63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7da4634985964688b08c580e06eb6c63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da4634985964688b08c580e06eb6c63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35871 } } }
I20260926 03:06:18.069103 3422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7da4634985964688b08c580e06eb6c63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7da4634985964688b08c580e06eb6c63. Latest consensus state: current_term: 1 leader_uuid: "7da4634985964688b08c580e06eb6c63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da4634985964688b08c580e06eb6c63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35871 } } }
I20260926 03:06:18.069448 3422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7da4634985964688b08c580e06eb6c63 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:18.069448 3421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7da4634985964688b08c580e06eb6c63 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:18.080355 3430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:06:18.085469 3430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:06:18.096141 3430 catalog_manager.cc:1347] Generated new cluster ID: 8c0fe569e5c64a64a3bb4ec233157439
I20260926 03:06:18.096277 3430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:06:18.100189 3417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:18.091251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53770 (local address 127.0.97.62:35871)
0926 03:06:18.091458 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:18.091465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:18.092112 (+ 647us) server_negotiation.cc:407] Connection header received
0926 03:06:18.092321 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:18.092328 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:18.092417 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:18.092546 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:18.093799 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:18.094644 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:18.096119 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:18.096759 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:18.096859 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:18.098308 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:18.098315 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:18.098320 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:06:18.099181 (+ 861us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:18.099272 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:06:18.099302 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:18.099424 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:18.099524 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 03:06:18.099624 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 03:06:18.109560 3430 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:06:18.111121 3430 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:06:18.118373 3430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7da4634985964688b08c580e06eb6c63: Generated new TSK 0
I20260926 03:06:18.119298 3430 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:06:18.127170 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 3347
I20260926 03:06:18.128800 3409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:18.216926 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 3347
2026-09-26T03:06:18Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1896 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:06:18 dist-test-slave-hmbw krb5kdc[3448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:06:18 dist-test-slave-hmbw krb5kdc[3448](info): set up 2 sockets
Sep 26 03:06:18 dist-test-slave-hmbw krb5kdc[3448](info): commencing operation
krb5kdc: starting...
W20260926 03:06:19.218681 388 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:06:19 dist-test-slave-hmbw krb5kdc[3448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:06:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:06:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:06:19.407105 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44721
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:19.723100 3461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:19.723448 3461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:19.723585 3461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:19.730942 3461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:06:19.731062 3461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:19.731107 3461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:19.731164 3461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:06:19.731225 3461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:06:19.741843 3461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:19.743369 3461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:19.745002 3461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:19.757340 3466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:19.757272 3467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:19.757308 3469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:19.758036 3461 server_base.cc:1034] running on GCE node
I20260926 03:06:19.759035 3461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:19.760639 3461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:19.761885 3461 hybrid_clock.cc:648] HybridClock initialized: now 1790391979761832 us; error 54 us; skew 500 ppm
I20260926 03:06:19.762290 3461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:19 dist-test-slave-hmbw krb5kdc[3448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:19.765381 3461 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:06:19.767015 3461 webserver.cc:460] Webserver started at http://127.0.97.62:40585/ using document root <none> and password file <none>
I20260926 03:06:19.767751 3461 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:19.767872 3461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:19.768162 3461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:19.771332 3461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "6db74aae815a49f2abff398f67d5a5f4"
format_stamp: "Formatted at 2026-09-26 03:06:19 on dist-test-slave-hmbw"
server_key: "5d2dca083d420629a6f8e792cb0133ba"
server_key_iv: "7ef430d66869cec3e8da86348938b154"
server_key_version: "encryptionkey@0"
I20260926 03:06:19.772113 3461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "6db74aae815a49f2abff398f67d5a5f4"
format_stamp: "Formatted at 2026-09-26 03:06:19 on dist-test-slave-hmbw"
server_key: "5d2dca083d420629a6f8e792cb0133ba"
server_key_iv: "7ef430d66869cec3e8da86348938b154"
server_key_version: "encryptionkey@0"
I20260926 03:06:19.777597 3461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 03:06:19.781472 3476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:19.782971 3461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:06:19.783169 3461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "6db74aae815a49f2abff398f67d5a5f4"
format_stamp: "Formatted at 2026-09-26 03:06:19 on dist-test-slave-hmbw"
server_key: "5d2dca083d420629a6f8e792cb0133ba"
server_key_iv: "7ef430d66869cec3e8da86348938b154"
server_key_version: "encryptionkey@0"
I20260926 03:06:19.783397 3461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:19.807802 3461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:19.812494 3461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:19.812791 3461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:19.831058 3461 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44721
I20260926 03:06:19.831077 3527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44721 every 8 connection(s)
I20260926 03:06:19.832437 3461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:06:19.835278 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 3461
I20260926 03:06:19.835477 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:06:19.836099 388 external_mini_cluster.cc:1428] Setting key 7707e02217682c038cd2cdb8e12b1990
I20260926 03:06:19.836366 3528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:19.850598 3528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6db74aae815a49f2abff398f67d5a5f4: Bootstrap starting.
Sep 26 03:06:19 dist-test-slave-hmbw krb5kdc[3448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:19.854409 3528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6db74aae815a49f2abff398f67d5a5f4: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:19.855825 3528 log.cc:824] T 00000000000000000000000000000000 P 6db74aae815a49f2abff398f67d5a5f4: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:19.859537 3528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6db74aae815a49f2abff398f67d5a5f4: No bootstrap required, opened a new log
I20260926 03:06:19.861274 3531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:19.838198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40884 (local address 127.0.97.62:44721)
0926 03:06:19.839778 (+ 1580us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:19.839793 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:19.839845 (+ 52us) server_negotiation.cc:407] Connection header received
0926 03:06:19.841393 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:19.841424 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:19.841994 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:19.842472 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:19.843723 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:19.845596 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:19.846980 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:19.848056 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:19.848265 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:19.852469 (+ 4204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:19.852482 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:19.852503 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:19.852552 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:19.856542 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:19.857537 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:19.857546 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:19.857554 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:19.857798 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:19.858146 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:19.858151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:19.858153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:19.858869 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:19.859172 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:19.859462 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 03:06:19.859922 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1318,"thread_start_us":1050,"threads_started":1}
I20260926 03:06:19.868067 3528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6db74aae815a49f2abff398f67d5a5f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6db74aae815a49f2abff398f67d5a5f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44721 } }
I20260926 03:06:19.868467 3528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6db74aae815a49f2abff398f67d5a5f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:19.868546 3528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6db74aae815a49f2abff398f67d5a5f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6db74aae815a49f2abff398f67d5a5f4, State: Initialized, Role: FOLLOWER
I20260926 03:06:19.869163 3528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6db74aae815a49f2abff398f67d5a5f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6db74aae815a49f2abff398f67d5a5f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44721 } }
I20260926 03:06:19.869381 3528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6db74aae815a49f2abff398f67d5a5f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:06:19.869481 3528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6db74aae815a49f2abff398f67d5a5f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:06:19.869606 3528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6db74aae815a49f2abff398f67d5a5f4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:19.872195 3528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6db74aae815a49f2abff398f67d5a5f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6db74aae815a49f2abff398f67d5a5f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44721 } }
I20260926 03:06:19.872699 3528 leader_election.cc:302] T 00000000000000000000000000000000 P 6db74aae815a49f2abff398f67d5a5f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6db74aae815a49f2abff398f67d5a5f4; no voters:
I20260926 03:06:19.873274 3528 leader_election.cc:288] T 00000000000000000000000000000000 P 6db74aae815a49f2abff398f67d5a5f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:06:19.873415 3533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6db74aae815a49f2abff398f67d5a5f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:19.874589 3533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6db74aae815a49f2abff398f67d5a5f4 [term 1 LEADER]: Becoming Leader. State: Replica: 6db74aae815a49f2abff398f67d5a5f4, State: Running, Role: LEADER
I20260926 03:06:19.875124 3533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6db74aae815a49f2abff398f67d5a5f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6db74aae815a49f2abff398f67d5a5f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44721 } }
I20260926 03:06:19.875500 3528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6db74aae815a49f2abff398f67d5a5f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:06:19.878294 3534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6db74aae815a49f2abff398f67d5a5f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6db74aae815a49f2abff398f67d5a5f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6db74aae815a49f2abff398f67d5a5f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44721 } } }
I20260926 03:06:19.878348 3535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6db74aae815a49f2abff398f67d5a5f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6db74aae815a49f2abff398f67d5a5f4. Latest consensus state: current_term: 1 leader_uuid: "6db74aae815a49f2abff398f67d5a5f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6db74aae815a49f2abff398f67d5a5f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44721 } } }
I20260926 03:06:19.878710 3534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6db74aae815a49f2abff398f67d5a5f4 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:19.878726 3535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6db74aae815a49f2abff398f67d5a5f4 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:19.880501 3541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:06:19.885778 3541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:06:19.895308 3541 catalog_manager.cc:1347] Generated new cluster ID: 20e628123e3343b6b6d43ed96439a019
I20260926 03:06:19.895438 3541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:06:19.914718 3541 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:06:19.916163 3541 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:06:19.922300 3541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6db74aae815a49f2abff398f67d5a5f4: Generated new TSK 0
I20260926 03:06:19.923321 3541 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:06:19.981642 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44721
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:20.335166 3554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:20.335490 3554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:20.335604 3554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:20.343124 3554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:20.343251 3554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:20.343405 3554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:06:20.354115 3554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:20.355607 3554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:20.356995 3554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:20.371120 3559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:20.371100 3560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:20.371165 3562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:20.371966 3554 server_base.cc:1034] running on GCE node
I20260926 03:06:20.372627 3554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:20.373667 3554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:20.374914 3554 hybrid_clock.cc:648] HybridClock initialized: now 1790391980374847 us; error 62 us; skew 500 ppm
I20260926 03:06:20.375314 3554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:20 dist-test-slave-hmbw krb5kdc[3448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:20.378962 3554 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:06:20.380609 3554 webserver.cc:460] Webserver started at http://127.0.97.1:39525/ using document root <none> and password file <none>
I20260926 03:06:20.381393 3554 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:20.381511 3554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:20.381806 3554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:20.384850 3554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/data/instance:
uuid: "7c64a70971c249e8b09d8d4f341f3b9b"
format_stamp: "Formatted at 2026-09-26 03:06:20 on dist-test-slave-hmbw"
server_key: "06b4b108776461f3ff8a3312225f1a48"
server_key_iv: "ff90c857a6a48660d62dc6f708317b1c"
server_key_version: "encryptionkey@0"
I20260926 03:06:20.385656 3554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7c64a70971c249e8b09d8d4f341f3b9b"
format_stamp: "Formatted at 2026-09-26 03:06:20 on dist-test-slave-hmbw"
server_key: "06b4b108776461f3ff8a3312225f1a48"
server_key_iv: "ff90c857a6a48660d62dc6f708317b1c"
server_key_version: "encryptionkey@0"
I20260926 03:06:20.391194 3554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 03:06:20.394869 3569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:20.396512 3554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:06:20.396703 3554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "7c64a70971c249e8b09d8d4f341f3b9b"
format_stamp: "Formatted at 2026-09-26 03:06:20 on dist-test-slave-hmbw"
server_key: "06b4b108776461f3ff8a3312225f1a48"
server_key_iv: "ff90c857a6a48660d62dc6f708317b1c"
server_key_version: "encryptionkey@0"
I20260926 03:06:20.396895 3554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:20.413296 3554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:20.418057 3554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:20.418387 3554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:20.419456 3554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:06:20.420902 3554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:20.420986 3554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:20.421114 3554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:20.421180 3554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:20.452116 3554 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37899
I20260926 03:06:20.452193 3681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37899 every 8 connection(s)
I20260926 03:06:20.453389 3554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:06:20.461097 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 3554
I20260926 03:06:20.461292 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:06:20.462067 388 external_mini_cluster.cc:1428] Setting key 2c9e9b225d4e4bd9d5a0193808753062
Sep 26 03:06:20 dist-test-slave-hmbw krb5kdc[3448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:20.474452 3684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:20.456124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48005 (local address 127.0.97.62:44721)
0926 03:06:20.456582 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:20.456589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:20.457577 (+ 988us) server_negotiation.cc:407] Connection header received
0926 03:06:20.459482 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:20.459487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:20.459574 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:20.459685 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:20.461316 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:20.462389 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:20.464145 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:20.464692 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:20.464833 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:20.468400 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:20.468407 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:20.468412 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:20.468454 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:20.471332 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:20.472140 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:20.472145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:20.472148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:20.472286 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:20.472682 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:20.472686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:20.472689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:20.473015 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:20.473157 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:20.473889 (+ 732us) server_negotiation.cc:299] Negotiation successful
0926 03:06:20.474115 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":200,"threads_started":1}
I20260926 03:06:20.475482 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:20.456689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44721 (local address 127.0.97.1:48005)
0926 03:06:20.457412 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:20.457456 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:20.459144 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:20.459904 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:20.459913 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:20.460517 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:20.461105 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:20.461122 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:20.462581 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:20.462585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:20.464005 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:20.464019 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:20.464918 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:20.464923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:20.465143 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:20.465867 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:20.465891 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:20.468129 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:20.471530 (+ 3401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:20.471539 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:20.471555 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:20.472002 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:20.472380 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:20.472383 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:20.472386 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:20.472582 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:20.473123 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:20.473129 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:20.473507 (+ 378us) client_negotiation.cc:769] Sending connection context
0926 03:06:20.473747 (+ 240us) client_negotiation.cc:240] Negotiation successful
0926 03:06:20.474151 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":284,"threads_started":1}
I20260926 03:06:20.477290 3682 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44721
I20260926 03:06:20.477648 3682 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:20.478413 3682 heartbeater.cc:511] Master 127.0.97.62:44721 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:06:20.480873 3493 ts_manager.cc:194] Registered new tserver with Master: 7c64a70971c249e8b09d8d4f341f3b9b (127.0.97.1:37899)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 03:06:20.482456 3493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:06:20.506592 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44721
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:20.830154 3688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:20.830490 3688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:20.830663 3688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:20.838274 3688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:20.838415 3688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:20.838608 3688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:06:20.848934 3688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:20.850351 3688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:20.851758 3688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:20.864715 3693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:20.864766 3696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:20.864786 3694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:20.865556 3688 server_base.cc:1034] running on GCE node
I20260926 03:06:20.866187 3688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:20.867174 3688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:20.868387 3688 hybrid_clock.cc:648] HybridClock initialized: now 1790391980868317 us; error 61 us; skew 500 ppm
I20260926 03:06:20.868713 3688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:20 dist-test-slave-hmbw krb5kdc[3448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:20.871882 3688 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:06:20.873344 3688 webserver.cc:460] Webserver started at http://127.0.97.2:44337/ using document root <none> and password file <none>
I20260926 03:06:20.874053 3688 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:20.874167 3688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:20.874464 3688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:20.877619 3688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/data/instance:
uuid: "6bb3fc28909a4195ba6ee89ff8b0a6f6"
format_stamp: "Formatted at 2026-09-26 03:06:20 on dist-test-slave-hmbw"
server_key: "2987a9071f7bd7d10160f49a14f68c7a"
server_key_iv: "47fb96f2a0833dc9c755fe712eed4500"
server_key_version: "encryptionkey@0"
I20260926 03:06:20.878476 3688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6bb3fc28909a4195ba6ee89ff8b0a6f6"
format_stamp: "Formatted at 2026-09-26 03:06:20 on dist-test-slave-hmbw"
server_key: "2987a9071f7bd7d10160f49a14f68c7a"
server_key_iv: "47fb96f2a0833dc9c755fe712eed4500"
server_key_version: "encryptionkey@0"
I20260926 03:06:20.884049 3688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 03:06:20.887987 3703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:20.889622 3688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 03:06:20.889807 3688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "6bb3fc28909a4195ba6ee89ff8b0a6f6"
format_stamp: "Formatted at 2026-09-26 03:06:20 on dist-test-slave-hmbw"
server_key: "2987a9071f7bd7d10160f49a14f68c7a"
server_key_iv: "47fb96f2a0833dc9c755fe712eed4500"
server_key_version: "encryptionkey@0"
I20260926 03:06:20.890000 3688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:20.903666 3688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:20.908393 3688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:20.908694 3688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:20.909711 3688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:06:20.911129 3688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:20.911208 3688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:20.911325 3688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:20.911376 3688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:20.942090 3688 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35053
I20260926 03:06:20.942123 3815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35053 every 8 connection(s)
I20260926 03:06:20.943424 3688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:06:20.944376 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 3688
I20260926 03:06:20.944550 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:06:20.945063 388 external_mini_cluster.cc:1428] Setting key 03ad832d3551fdfb2b4adeb03edca650
Sep 26 03:06:20 dist-test-slave-hmbw krb5kdc[3448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 03:06:20.964076 3684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:20.946288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40621 (local address 127.0.97.62:44721)
0926 03:06:20.946467 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:20.946472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:20.947859 (+ 1387us) server_negotiation.cc:407] Connection header received
0926 03:06:20.949691 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:20.949706 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:20.949791 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:20.949875 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:20.951567 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:20.952359 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:20.954244 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:20.954683 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:20.954801 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:20.958362 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:20.958367 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:20.958370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:20.958407 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:20.960827 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:20.961645 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:20.961650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:20.961652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:20.961781 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:20.962238 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:20.962242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:20.962245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:20.962540 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:20.962669 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:20.963452 (+ 783us) server_negotiation.cc:299] Negotiation successful
0926 03:06:20.963742 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:06:20.965085 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:20.946864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44721 (local address 127.0.97.2:40621)
0926 03:06:20.947683 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:20.947737 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:20.949407 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:20.950124 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:20.950133 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:20.950780 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:20.951384 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:20.951404 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:20.952564 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:20.952571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:20.954082 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:20.954100 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:20.954933 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:20.954941 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:20.955158 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:20.955815 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:20.955839 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:20.958101 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:20.960988 (+ 2887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:20.960997 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:20.961019 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:20.961510 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:20.961892 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:20.961896 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:20.961900 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:20.962120 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:20.962668 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:20.962678 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:20.963064 (+ 386us) client_negotiation.cc:769] Sending connection context
0926 03:06:20.963346 (+ 282us) client_negotiation.cc:240] Negotiation successful
0926 03:06:20.963906 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":331,"threads_started":1}
I20260926 03:06:20.966775 3816 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44721
I20260926 03:06:20.967193 3816 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:20.968142 3816 heartbeater.cc:511] Master 127.0.97.62:44721 requested a full tablet report, sending...
I20260926 03:06:20.970165 3493 ts_manager.cc:194] Registered new tserver with Master: 6bb3fc28909a4195ba6ee89ff8b0a6f6 (127.0.97.2:35053)
I20260926 03:06:20.971084 3493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:06:20.991361 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44721
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:21.315420 3821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:21.315721 3821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:21.315814 3821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:21.323491 3821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:21.323617 3821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:21.323731 3821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:06:21.333665 3821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:21.334951 3821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:21.336263 3821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:21.349593 3826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:21.349570 3829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:21.349534 3827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:21.350524 3821 server_base.cc:1034] running on GCE node
I20260926 03:06:21.351163 3821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:21.352159 3821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:21.353363 3821 hybrid_clock.cc:648] HybridClock initialized: now 1790391981353331 us; error 58 us; skew 500 ppm
I20260926 03:06:21.353688 3821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:21 dist-test-slave-hmbw krb5kdc[3448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:21.357012 3821 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:06:21.358641 3821 webserver.cc:460] Webserver started at http://127.0.97.3:34983/ using document root <none> and password file <none>
I20260926 03:06:21.359323 3821 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:21.359498 3821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:21.359776 3821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:21.362787 3821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/data/instance:
uuid: "227eeff7a44547a9a3ec31ced4b8112c"
format_stamp: "Formatted at 2026-09-26 03:06:21 on dist-test-slave-hmbw"
server_key: "fed0649e3c45819f55b97f8c03c7b768"
server_key_iv: "7c0a6140739db9933ba9fa8263bd2419"
server_key_version: "encryptionkey@0"
I20260926 03:06:21.363545 3821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/wal/instance:
uuid: "227eeff7a44547a9a3ec31ced4b8112c"
format_stamp: "Formatted at 2026-09-26 03:06:21 on dist-test-slave-hmbw"
server_key: "fed0649e3c45819f55b97f8c03c7b768"
server_key_iv: "7c0a6140739db9933ba9fa8263bd2419"
server_key_version: "encryptionkey@0"
I20260926 03:06:21.368762 3821 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:06:21.372372 3836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:21.373750 3821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:06:21.373888 3821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "227eeff7a44547a9a3ec31ced4b8112c"
format_stamp: "Formatted at 2026-09-26 03:06:21 on dist-test-slave-hmbw"
server_key: "fed0649e3c45819f55b97f8c03c7b768"
server_key_iv: "7c0a6140739db9933ba9fa8263bd2419"
server_key_version: "encryptionkey@0"
I20260926 03:06:21.374094 3821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:21.407477 3821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:21.411773 3821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:21.412051 3821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:21.413012 3821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:06:21.414451 3821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:21.414551 3821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260926 03:06:21.414682 3821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:21.414731 3821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:21.445899 3821 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46075
I20260926 03:06:21.445928 3948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46075 every 8 connection(s)
I20260926 03:06:21.447216 3821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:06:21.450196 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 3821
I20260926 03:06:21.450366 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391950058021-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:06:21.450888 388 external_mini_cluster.cc:1428] Setting key d4fa4eb4166fabb57f9355a629ed9d42
Sep 26 03:06:21 dist-test-slave-hmbw krb5kdc[3448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:21.472647 3684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:21.449898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50721 (local address 127.0.97.62:44721)
0926 03:06:21.450114 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:21.450120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:21.451570 (+ 1450us) server_negotiation.cc:407] Connection header received
0926 03:06:21.457132 (+ 5562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:21.457136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:21.457212 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:21.457315 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:21.459787 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:21.460564 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:21.462253 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:21.462736 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:21.462845 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:21.466456 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:21.466462 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:21.466466 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:21.466535 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:21.469567 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:21.470275 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:21.470279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:21.470281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:21.470407 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:21.470897 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:21.470900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:21.470902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:21.471190 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:21.471309 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:21.472132 (+ 823us) server_negotiation.cc:299] Negotiation successful
0926 03:06:21.472320 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 03:06:21.473587 3951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:21.450544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44721 (local address 127.0.97.3:50721)
0926 03:06:21.451374 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:21.451431 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:21.456742 (+ 5311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:21.457599 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:21.457612 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:21.458342 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:21.459602 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:21.459622 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:21.460725 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:21.460729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:21.462020 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:21.462034 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:21.462932 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:21.462937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:21.463141 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:21.463802 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:21.463830 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:21.466242 (+ 2412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:21.469712 (+ 3470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:21.469720 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:21.469736 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:21.470173 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:21.470561 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:21.470567 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:21.470570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:21.470789 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:21.471395 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:21.471402 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:21.471759 (+ 357us) client_negotiation.cc:769] Sending connection context
0926 03:06:21.472005 (+ 246us) client_negotiation.cc:240] Negotiation successful
0926 03:06:21.472455 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":315,"threads_started":1}
I20260926 03:06:21.475536 3949 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44721
I20260926 03:06:21.475850 3949 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:21.476727 3949 heartbeater.cc:511] Master 127.0.97.62:44721 requested a full tablet report, sending...
I20260926 03:06:21.478536 3492 ts_manager.cc:194] Registered new tserver with Master: 227eeff7a44547a9a3ec31ced4b8112c (127.0.97.3:46075)
I20260926 03:06:21.479432 3492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50721
I20260926 03:06:21.485647 3682 heartbeater.cc:503] Master 127.0.97.62:44721 was elected leader, sending a full tablet report...
I20260926 03:06:21.490476 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 03:06:21 dist-test-slave-hmbw krb5kdc[3448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391981, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 03:06:21 dist-test-slave-hmbw krb5kdc[3448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391981, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:21.579130 3684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:21.562980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40886 (local address 127.0.97.62:44721)
0926 03:06:21.563180 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:21.563187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:21.563694 (+ 507us) server_negotiation.cc:407] Connection header received
0926 03:06:21.563825 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:21.563831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:21.563907 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:21.564017 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:21.565401 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:21.566122 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:21.567676 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:21.568163 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:21.568388 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:21.573076 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:21.573082 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:21.573086 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:21.573126 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:21.575856 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:21.576966 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:21.576973 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:21.576976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:21.577154 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:21.577577 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:21.577583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:21.577586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:21.577988 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:21.578222 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:21.578592 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 03:06:21.578822 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 03:06:21.579730 3492 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40886
I20260926 03:06:21.582101 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 3554
Sep 26 03:06:21 dist-test-slave-hmbw krb5kdc[3448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391981, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:06:21.598136 3960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:21.583083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42138 (local address 127.0.97.1:37899)
0926 03:06:21.583699 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:21.583707 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:21.583732 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:06:21.583790 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:21.583796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:21.584009 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:21.584139 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:21.585397 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:21.586487 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:21.587883 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:21.588514 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:21.588623 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:21.592522 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:21.592537 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:21.592553 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:21.592592 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:21.595553 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:21.596272 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:21.596279 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:21.596281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:21.596418 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:21.596793 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:21.596796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:21.596798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:21.597140 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:21.597363 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:21.597624 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 03:06:21.597821 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":300,"threads_started":1}
W20260926 03:06:21.599422 3677 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42138
W20260926 03:06:21.600569 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 3554: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:06:21.600755 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 3554
I20260926 03:06:21.613611 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 3688
Sep 26 03:06:21 dist-test-slave-hmbw krb5kdc[3448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391981, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:06:21.629452 3961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:21.614461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57272 (local address 127.0.97.2:35053)
0926 03:06:21.614982 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:21.614989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:21.615023 (+ 34us) server_negotiation.cc:407] Connection header received
0926 03:06:21.615083 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:21.615089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:21.615305 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:21.615441 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:21.616724 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:21.617730 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:21.618978 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:21.619610 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:21.619715 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:21.623750 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:21.623765 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:21.623781 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:21.623830 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:21.626634 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:21.627282 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:21.627289 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:21.627291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:21.627466 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:21.627883 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:21.627888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:21.627890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:21.628345 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:21.628596 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:21.628889 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 03:06:21.629098 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":249,"threads_started":1}
W20260926 03:06:21.630746 3811 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57272
W20260926 03:06:21.631855 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 3688: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:06:21.631996 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 3688
I20260926 03:06:21.644992 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 3821
Sep 26 03:06:21 dist-test-slave-hmbw krb5kdc[3448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391981, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:06:21.662343 3962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:21.645960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41900 (local address 127.0.97.3:46075)
0926 03:06:21.646597 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:21.646606 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:21.646640 (+ 34us) server_negotiation.cc:407] Connection header received
0926 03:06:21.646712 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:21.646721 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:21.646995 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:21.647150 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:21.648515 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:21.649804 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:21.651097 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:21.651689 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:21.651820 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:21.656105 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:21.656119 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:21.656134 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:21.656172 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:21.659218 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:21.660110 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:21.660119 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:21.660126 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:21.660280 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:21.660755 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:21.660759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:21.660761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:21.661154 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:21.661460 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:21.661690 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 03:06:21.661912 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":322,"threads_started":1}
W20260926 03:06:21.663679 3944 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41900
W20260926 03:06:21.664803 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 3821: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:06:21.664973 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 3821
I20260926 03:06:21.677865 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 3461
I20260926 03:06:21.679116 3523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:21.791102 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 3461
2026-09-26T03:06:21Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3579 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:06:21 dist-test-slave-hmbw krb5kdc[3966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:06:21 dist-test-slave-hmbw krb5kdc[3966](info): set up 2 sockets
Sep 26 03:06:21 dist-test-slave-hmbw krb5kdc[3966](info): commencing operation
krb5kdc: starting...
W20260926 03:06:22.914789 388 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:06:23 dist-test-slave-hmbw krb5kdc[3966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:06:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:06:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:06:23.108148 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43431
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43431
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:23.437151 3979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:23.437472 3979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:23.437561 3979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:23.444931 3979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:06:23.445046 3979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:23.445101 3979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:23.445154 3979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:06:23.445196 3979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:06:23.455137 3979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:23.456450 3979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:23.457726 3979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:23.469720 3984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:23.469811 3987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:23.469779 3985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:23.470841 3979 server_base.cc:1034] running on GCE node
I20260926 03:06:23.471648 3979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:23.473202 3979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:23.474428 3979 hybrid_clock.cc:648] HybridClock initialized: now 1790391983474344 us; error 74 us; skew 500 ppm
I20260926 03:06:23.474848 3979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:23 dist-test-slave-hmbw krb5kdc[3966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:23.484978 3979 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:06:23.486596 3979 webserver.cc:460] Webserver started at http://127.0.97.62:35673/ using document root <none> and password file <none>
I20260926 03:06:23.487293 3979 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:23.487401 3979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:23.487681 3979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:23.490712 3979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "d21af2ce758847c3b2ab05a92a5bfaf0"
format_stamp: "Formatted at 2026-09-26 03:06:23 on dist-test-slave-hmbw"
server_key: "52d338b9962d28d5bd8ad9ab28b884d0"
server_key_iv: "5f9be5dc6a1929f41beabdd01bc62ed1"
server_key_version: "encryptionkey@0"
I20260926 03:06:23.491466 3979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "d21af2ce758847c3b2ab05a92a5bfaf0"
format_stamp: "Formatted at 2026-09-26 03:06:23 on dist-test-slave-hmbw"
server_key: "52d338b9962d28d5bd8ad9ab28b884d0"
server_key_iv: "5f9be5dc6a1929f41beabdd01bc62ed1"
server_key_version: "encryptionkey@0"
I20260926 03:06:23.496696 3979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:06:23.500500 3994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:23.502101 3979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:06:23.502267 3979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "d21af2ce758847c3b2ab05a92a5bfaf0"
format_stamp: "Formatted at 2026-09-26 03:06:23 on dist-test-slave-hmbw"
server_key: "52d338b9962d28d5bd8ad9ab28b884d0"
server_key_iv: "5f9be5dc6a1929f41beabdd01bc62ed1"
server_key_version: "encryptionkey@0"
I20260926 03:06:23.502454 3979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:23.536521 3979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:23.540747 3979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:23.540992 3979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:23.558125 3979 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43431
I20260926 03:06:23.558157 4045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43431 every 8 connection(s)
I20260926 03:06:23.559588 3979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:06:23.563490 4046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:23.567008 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 3979
I20260926 03:06:23.567200 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:06:23.567811 388 external_mini_cluster.cc:1428] Setting key 78f91293bc0702ff97a0f3810292aefa
I20260926 03:06:23.577067 4046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d21af2ce758847c3b2ab05a92a5bfaf0: Bootstrap starting.
I20260926 03:06:23.580806 4046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d21af2ce758847c3b2ab05a92a5bfaf0: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:23.582252 4046 log.cc:824] T 00000000000000000000000000000000 P d21af2ce758847c3b2ab05a92a5bfaf0: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:06:23 dist-test-slave-hmbw krb5kdc[3966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:23.585879 4046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d21af2ce758847c3b2ab05a92a5bfaf0: No bootstrap required, opened a new log
I20260926 03:06:23.594695 4049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:23.569953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55894 (local address 127.0.97.62:43431)
0926 03:06:23.570767 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:23.570783 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:23.570836 (+ 53us) server_negotiation.cc:407] Connection header received
0926 03:06:23.572652 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:23.572687 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:23.573297 (+ 610us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:23.573790 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:23.575300 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:23.577273 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:23.578811 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:23.579870 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:23.580120 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:23.585300 (+ 5180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:23.585309 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:23.585333 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:23.585375 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:23.589300 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:23.590282 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:23.590293 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:23.590303 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:23.590545 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:23.590959 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:23.590964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:23.590967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:23.591744 (+ 777us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:23.592057 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:23.592490 (+ 433us) server_negotiation.cc:299] Negotiation successful
0926 03:06:23.593183 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":332,"threads_started":1}
I20260926 03:06:23.596335 4046 raft_consensus.cc:348] T 00000000000000000000000000000000 P d21af2ce758847c3b2ab05a92a5bfaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21af2ce758847c3b2ab05a92a5bfaf0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43431 } }
I20260926 03:06:23.596724 4046 raft_consensus.cc:374] T 00000000000000000000000000000000 P d21af2ce758847c3b2ab05a92a5bfaf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:23.596829 4046 raft_consensus.cc:729] T 00000000000000000000000000000000 P d21af2ce758847c3b2ab05a92a5bfaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d21af2ce758847c3b2ab05a92a5bfaf0, State: Initialized, Role: FOLLOWER
I20260926 03:06:23.597509 4046 consensus_queue.cc:260] T 00000000000000000000000000000000 P d21af2ce758847c3b2ab05a92a5bfaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21af2ce758847c3b2ab05a92a5bfaf0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43431 } }
I20260926 03:06:23.597815 4046 raft_consensus.cc:388] T 00000000000000000000000000000000 P d21af2ce758847c3b2ab05a92a5bfaf0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:06:23.597951 4046 raft_consensus.cc:482] T 00000000000000000000000000000000 P d21af2ce758847c3b2ab05a92a5bfaf0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:06:23.598121 4046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d21af2ce758847c3b2ab05a92a5bfaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:23.600440 4046 raft_consensus.cc:504] T 00000000000000000000000000000000 P d21af2ce758847c3b2ab05a92a5bfaf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21af2ce758847c3b2ab05a92a5bfaf0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43431 } }
I20260926 03:06:23.600925 4046 leader_election.cc:302] T 00000000000000000000000000000000 P d21af2ce758847c3b2ab05a92a5bfaf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d21af2ce758847c3b2ab05a92a5bfaf0; no voters:
I20260926 03:06:23.601603 4046 leader_election.cc:288] T 00000000000000000000000000000000 P d21af2ce758847c3b2ab05a92a5bfaf0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:06:23.601735 4051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d21af2ce758847c3b2ab05a92a5bfaf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:23.603119 4051 raft_consensus.cc:686] T 00000000000000000000000000000000 P d21af2ce758847c3b2ab05a92a5bfaf0 [term 1 LEADER]: Becoming Leader. State: Replica: d21af2ce758847c3b2ab05a92a5bfaf0, State: Running, Role: LEADER
I20260926 03:06:23.603672 4051 consensus_queue.cc:237] T 00000000000000000000000000000000 P d21af2ce758847c3b2ab05a92a5bfaf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21af2ce758847c3b2ab05a92a5bfaf0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43431 } }
I20260926 03:06:23.603802 4046 sys_catalog.cc:564] T 00000000000000000000000000000000 P d21af2ce758847c3b2ab05a92a5bfaf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:06:23.606276 4053 sys_catalog.cc:455] T 00000000000000000000000000000000 P d21af2ce758847c3b2ab05a92a5bfaf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d21af2ce758847c3b2ab05a92a5bfaf0. Latest consensus state: current_term: 1 leader_uuid: "d21af2ce758847c3b2ab05a92a5bfaf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21af2ce758847c3b2ab05a92a5bfaf0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43431 } } }
I20260926 03:06:23.606742 4053 sys_catalog.cc:458] T 00000000000000000000000000000000 P d21af2ce758847c3b2ab05a92a5bfaf0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:23.607249 4052 sys_catalog.cc:455] T 00000000000000000000000000000000 P d21af2ce758847c3b2ab05a92a5bfaf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d21af2ce758847c3b2ab05a92a5bfaf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d21af2ce758847c3b2ab05a92a5bfaf0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43431 } } }
I20260926 03:06:23.607583 4052 sys_catalog.cc:458] T 00000000000000000000000000000000 P d21af2ce758847c3b2ab05a92a5bfaf0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:23.610524 4060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:06:23.616496 4060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:06:23.628067 4060 catalog_manager.cc:1347] Generated new cluster ID: 32bce7ae235042abb434cb7a7d1c28b8
I20260926 03:06:23.628209 4060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:06:23.644781 4060 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:06:23.646313 4060 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:06:23.653445 4060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d21af2ce758847c3b2ab05a92a5bfaf0: Generated new TSK 0
I20260926 03:06:23.654320 4060 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:06:23.702378 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43431
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:24.022455 4072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:24.022799 4072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:24.022922 4072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:24.030169 4072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:24.030308 4072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:24.030454 4072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:06:24.041008 4072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:24.042363 4072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:24.043833 4072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:24.057035 4080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:24.057003 4077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:24.057066 4078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:24.057664 4072 server_base.cc:1034] running on GCE node
I20260926 03:06:24.058432 4072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:24.059516 4072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:24.060751 4072 hybrid_clock.cc:648] HybridClock initialized: now 1790391984060676 us; error 60 us; skew 500 ppm
I20260926 03:06:24.061106 4072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:24 dist-test-slave-hmbw krb5kdc[3966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:24.064261 4072 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:06:24.065992 4072 webserver.cc:460] Webserver started at http://127.0.97.1:45367/ using document root <none> and password file <none>
I20260926 03:06:24.066725 4072 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:24.066840 4072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:24.067121 4072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:24.069947 4072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-0/data/instance:
uuid: "a07a5337ade441f4a9362ec80ed009f8"
format_stamp: "Formatted at 2026-09-26 03:06:24 on dist-test-slave-hmbw"
server_key: "f0b21adc42b38c9496bb3f3b41596a2c"
server_key_iv: "0f08ba2bbedf4f7ad70ca0360f79a166"
server_key_version: "encryptionkey@0"
I20260926 03:06:24.070763 4072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a07a5337ade441f4a9362ec80ed009f8"
format_stamp: "Formatted at 2026-09-26 03:06:24 on dist-test-slave-hmbw"
server_key: "f0b21adc42b38c9496bb3f3b41596a2c"
server_key_iv: "0f08ba2bbedf4f7ad70ca0360f79a166"
server_key_version: "encryptionkey@0"
I20260926 03:06:24.075991 4072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260926 03:06:24.079734 4087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:24.081203 4072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:06:24.081367 4072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "a07a5337ade441f4a9362ec80ed009f8"
format_stamp: "Formatted at 2026-09-26 03:06:24 on dist-test-slave-hmbw"
server_key: "f0b21adc42b38c9496bb3f3b41596a2c"
server_key_iv: "0f08ba2bbedf4f7ad70ca0360f79a166"
server_key_version: "encryptionkey@0"
I20260926 03:06:24.081576 4072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:24.097342 4072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:24.101496 4072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:24.101747 4072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:24.102687 4072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:06:24.103928 4072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:24.104022 4072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:24.104140 4072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:24.104202 4072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:24.134151 4072 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33279
I20260926 03:06:24.134205 4199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33279 every 8 connection(s)
I20260926 03:06:24.135510 4072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:06:24.142442 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 4072
I20260926 03:06:24.142684 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:06:24.143428 388 external_mini_cluster.cc:1428] Setting key da9830f66899a6bebc9115116b734006
Sep 26 03:06:24 dist-test-slave-hmbw krb5kdc[3966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:24.156462 4202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:24.138171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37629 (local address 127.0.97.62:43431)
0926 03:06:24.138646 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:24.138652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:24.139556 (+ 904us) server_negotiation.cc:407] Connection header received
0926 03:06:24.141277 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:24.141281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:24.141355 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:24.141454 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:24.143282 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:24.144123 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:24.145985 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:24.146379 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:24.146549 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:24.150453 (+ 3904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:24.150460 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:24.150464 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:24.150539 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:24.153154 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:24.154096 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:24.154100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:24.154103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:24.154216 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:24.154706 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:24.154710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:24.154713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:24.155020 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:24.155136 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:24.155992 (+ 856us) server_negotiation.cc:299] Negotiation successful
0926 03:06:24.156175 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":215,"threads_started":1}
I20260926 03:06:24.157852 4203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:24.138683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43431 (local address 127.0.97.1:37629)
0926 03:06:24.139387 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:24.139429 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:24.141022 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:24.141666 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:24.141674 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:24.142259 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:24.143076 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:24.143102 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:24.144276 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:24.144281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:24.145825 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:24.145841 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:24.146648 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:24.146656 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:24.146963 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:24.147664 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:24.147690 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:24.150203 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:24.153352 (+ 3149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:24.153361 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:24.153379 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:24.153954 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:24.154308 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:24.154312 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:24.154315 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:24.154574 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:24.155148 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:24.155157 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:24.155564 (+ 407us) client_negotiation.cc:769] Sending connection context
0926 03:06:24.155852 (+ 288us) client_negotiation.cc:240] Negotiation successful
0926 03:06:24.156463 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":270,"threads_started":1}
I20260926 03:06:24.159505 4200 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43431
I20260926 03:06:24.159847 4200 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:24.160821 4200 heartbeater.cc:511] Master 127.0.97.62:43431 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:06:24.163074 4011 ts_manager.cc:194] Registered new tserver with Master: a07a5337ade441f4a9362ec80ed009f8 (127.0.97.1:33279)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 03:06:24.164561 4011 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:06:24.186260 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43431
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:24.503527 4206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:24.503841 4206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:24.503937 4206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:24.511695 4206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:24.511819 4206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:24.511930 4206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:06:24.521697 4206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:24.523044 4206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:24.524281 4206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:24.537114 4212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:24.537139 4211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:24.537081 4214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:24.537762 4206 server_base.cc:1034] running on GCE node
I20260926 03:06:24.538380 4206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:24.539377 4206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:24.540625 4206 hybrid_clock.cc:648] HybridClock initialized: now 1790391984540565 us; error 52 us; skew 500 ppm
I20260926 03:06:24.540971 4206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:24 dist-test-slave-hmbw krb5kdc[3966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:24.544219 4206 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:06:24.546110 4206 webserver.cc:460] Webserver started at http://127.0.97.2:41583/ using document root <none> and password file <none>
I20260926 03:06:24.546866 4206 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:24.546976 4206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:24.547274 4206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:24.550168 4206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-1/data/instance:
uuid: "3e4f25e575ef41b89c73c3b8ff92687e"
format_stamp: "Formatted at 2026-09-26 03:06:24 on dist-test-slave-hmbw"
server_key: "80312cf8be65c30d9ab2d499e526775c"
server_key_iv: "38377c7ceff0b52b3da61b65fd22c13e"
server_key_version: "encryptionkey@0"
I20260926 03:06:24.550959 4206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3e4f25e575ef41b89c73c3b8ff92687e"
format_stamp: "Formatted at 2026-09-26 03:06:24 on dist-test-slave-hmbw"
server_key: "80312cf8be65c30d9ab2d499e526775c"
server_key_iv: "38377c7ceff0b52b3da61b65fd22c13e"
server_key_version: "encryptionkey@0"
I20260926 03:06:24.556206 4206 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:06:24.560148 4221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:24.561828 4206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:06:24.562014 4206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "3e4f25e575ef41b89c73c3b8ff92687e"
format_stamp: "Formatted at 2026-09-26 03:06:24 on dist-test-slave-hmbw"
server_key: "80312cf8be65c30d9ab2d499e526775c"
server_key_iv: "38377c7ceff0b52b3da61b65fd22c13e"
server_key_version: "encryptionkey@0"
I20260926 03:06:24.562217 4206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:24.585000 4206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:24.589357 4206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:24.589636 4206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:24.590652 4206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:06:24.591961 4206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:24.592038 4206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:24.592146 4206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:24.592195 4206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:24.622327 4206 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42547
I20260926 03:06:24.622385 4333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42547 every 8 connection(s)
I20260926 03:06:24.623646 4206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:06:24.624609 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 4206
I20260926 03:06:24.624820 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:06:24.625332 388 external_mini_cluster.cc:1428] Setting key aa1b06d2944fe927b098feb3cf0c5d76
Sep 26 03:06:24 dist-test-slave-hmbw krb5kdc[3966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 03:06:24.646735 4202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:24.626711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53123 (local address 127.0.97.62:43431)
0926 03:06:24.626905 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:24.626911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:24.628474 (+ 1563us) server_negotiation.cc:407] Connection header received
0926 03:06:24.631000 (+ 2526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:24.631006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:24.631090 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:24.631181 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:24.633225 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:24.634090 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:24.635709 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:24.636288 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:24.636434 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:24.640053 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:24.640060 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:24.640065 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:24.640105 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:24.643534 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:24.644384 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:24.644391 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:24.644394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:24.644547 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:24.645094 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:24.645099 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:24.645101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:24.645393 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:24.645509 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:24.646269 (+ 760us) server_negotiation.cc:299] Negotiation successful
0926 03:06:24.646443 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 03:06:24.648056 4336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:24.627206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43431 (local address 127.0.97.2:53123)
0926 03:06:24.628287 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:24.628350 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:24.630658 (+ 2308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:24.631454 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:24.631466 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:24.632222 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:24.633002 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:24.633032 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:24.634240 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:24.634244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:24.635557 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:24.635569 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:24.636520 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:24.636525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:24.636753 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:24.637407 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:24.637430 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:24.639779 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:24.643719 (+ 3940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:24.643727 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:24.643742 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:24.644232 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:24.644711 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:24.644716 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:24.644720 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:24.644934 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:24.645500 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:24.645508 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:24.645889 (+ 381us) client_negotiation.cc:769] Sending connection context
0926 03:06:24.646153 (+ 264us) client_negotiation.cc:240] Negotiation successful
0926 03:06:24.646677 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":480,"threads_started":1}
I20260926 03:06:24.649683 4334 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43431
I20260926 03:06:24.650015 4334 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:24.650861 4334 heartbeater.cc:511] Master 127.0.97.62:43431 requested a full tablet report, sending...
I20260926 03:06:24.652515 4011 ts_manager.cc:194] Registered new tserver with Master: 3e4f25e575ef41b89c73c3b8ff92687e (127.0.97.2:42547)
I20260926 03:06:24.653340 4011 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:06:24.673331 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43431
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:24.990031 4339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:24.990324 4339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:24.990459 4339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:24.997794 4339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:24.997937 4339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:24.998090 4339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:06:25.008344 4339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:25.009678 4339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:25.011019 4339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:25.024214 4345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:25.024283 4344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:25.024231 4347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:25.025031 4339 server_base.cc:1034] running on GCE node
I20260926 03:06:25.025668 4339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:25.026755 4339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:25.027992 4339 hybrid_clock.cc:648] HybridClock initialized: now 1790391985027944 us; error 47 us; skew 500 ppm
I20260926 03:06:25.028318 4339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:25 dist-test-slave-hmbw krb5kdc[3966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:25.031431 4339 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:06:25.032927 4339 webserver.cc:460] Webserver started at http://127.0.97.3:34581/ using document root <none> and password file <none>
I20260926 03:06:25.033587 4339 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:25.033684 4339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:25.033950 4339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:25.036964 4339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-2/data/instance:
uuid: "b0965bc9ae0043ee8aca3f2e643cebf6"
format_stamp: "Formatted at 2026-09-26 03:06:25 on dist-test-slave-hmbw"
server_key: "e9092980f24a87f7ffbb13014b83bb8f"
server_key_iv: "3d2415a5b01550c8a746f37619788276"
server_key_version: "encryptionkey@0"
I20260926 03:06:25.037719 4339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b0965bc9ae0043ee8aca3f2e643cebf6"
format_stamp: "Formatted at 2026-09-26 03:06:25 on dist-test-slave-hmbw"
server_key: "e9092980f24a87f7ffbb13014b83bb8f"
server_key_iv: "3d2415a5b01550c8a746f37619788276"
server_key_version: "encryptionkey@0"
I20260926 03:06:25.043205 4339 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 03:06:25.046912 4354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:25.048353 4339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:06:25.048525 4339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "b0965bc9ae0043ee8aca3f2e643cebf6"
format_stamp: "Formatted at 2026-09-26 03:06:25 on dist-test-slave-hmbw"
server_key: "e9092980f24a87f7ffbb13014b83bb8f"
server_key_iv: "3d2415a5b01550c8a746f37619788276"
server_key_version: "encryptionkey@0"
I20260926 03:06:25.048735 4339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:25.091681 4339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:25.095974 4339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:25.096268 4339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:25.097246 4339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:06:25.098646 4339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:25.098716 4339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:25.098824 4339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:25.098870 4339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:25.128860 4339 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45947
I20260926 03:06:25.128916 4466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45947 every 8 connection(s)
I20260926 03:06:25.130100 4339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:06:25.131847 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 4339
I20260926 03:06:25.132016 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391950058021-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:06:25.132486 388 external_mini_cluster.cc:1428] Setting key c32303aad860adddd591392b61a991a5
Sep 26 03:06:25 dist-test-slave-hmbw krb5kdc[3966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:25.150384 4202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:25.132494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37775 (local address 127.0.97.62:43431)
0926 03:06:25.132679 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:25.132685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:25.134254 (+ 1569us) server_negotiation.cc:407] Connection header received
0926 03:06:25.136387 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:25.136392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:25.136491 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:25.136579 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:25.137963 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:25.138744 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:25.140220 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:25.140816 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:25.140935 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:25.144359 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:25.144365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:25.144368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:25.144402 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:25.146960 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:25.147926 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:25.147931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:25.147933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:25.148079 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:25.148680 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:25.148685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:25.148688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:25.149080 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:25.149200 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:25.149975 (+ 775us) server_negotiation.cc:299] Negotiation successful
0926 03:06:25.150147 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 03:06:25.151324 4469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:25.133068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43431 (local address 127.0.97.3:37775)
0926 03:06:25.134104 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:25.134149 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:25.136120 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:25.136808 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:25.136816 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:25.137295 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:25.137814 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:25.137829 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:25.138880 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:25.138884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:25.140077 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:25.140089 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:25.141035 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:25.141042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:25.141329 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:25.141955 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:25.141978 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:25.144151 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:25.147134 (+ 2983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:25.147144 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:25.147164 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:25.147818 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:25.148213 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:25.148219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:25.148230 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:25.148563 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:25.149206 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:25.149213 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:25.149593 (+ 380us) client_negotiation.cc:769] Sending connection context
0926 03:06:25.149828 (+ 235us) client_negotiation.cc:240] Negotiation successful
0926 03:06:25.150226 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":559,"threads_started":1}
I20260926 03:06:25.152599 4467 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43431
I20260926 03:06:25.152891 4467 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:25.153712 4467 heartbeater.cc:511] Master 127.0.97.62:43431 requested a full tablet report, sending...
I20260926 03:06:25.155019 4011 ts_manager.cc:194] Registered new tserver with Master: b0965bc9ae0043ee8aca3f2e643cebf6 (127.0.97.3:45947)
I20260926 03:06:25.155792 4011 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37775
I20260926 03:06:25.161604 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:06:25.167503 4200 heartbeater.cc:503] Master 127.0.97.62:43431 was elected leader, sending a full tablet report...
Sep 26 03:06:25 dist-test-slave-hmbw krb5kdc[3966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 03:06:25 dist-test-slave-hmbw krb5kdc[3966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:06:25.209897 4474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:25.193339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51344 (local address 127.0.97.1:33279)
0926 03:06:25.193952 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:25.193959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:25.194005 (+ 46us) server_negotiation.cc:407] Connection header received
0926 03:06:25.194064 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:25.194070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:25.194272 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:25.194398 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:25.195851 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:25.196931 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:25.198193 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:25.198841 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:25.198948 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:25.203437 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:25.203455 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:25.203474 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:25.203522 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:25.206740 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:25.207573 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:25.207582 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:25.207585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:25.207755 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:25.208220 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:25.208224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:25.208227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:25.208625 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:25.208922 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:25.209256 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 03:06:25.209475 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":319,"threads_started":1}
Sep 26 03:06:25 dist-test-slave-hmbw krb5kdc[3966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:25.228209 4202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:25.214294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41338 (local address 127.0.97.62:43431)
0926 03:06:25.214555 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:25.214562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:25.214746 (+ 184us) server_negotiation.cc:407] Connection header received
0926 03:06:25.214912 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:25.214918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:25.214988 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:25.215071 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:25.216270 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:25.217049 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:25.218823 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:25.219266 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:25.219384 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:25.223325 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:25.223330 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:25.223334 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:25.223376 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:25.225736 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:25.226343 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:25.226346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:25.226349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:25.226462 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:25.226898 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:25.226902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:25.226904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:25.227238 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:25.227419 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:25.227740 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 03:06:25.227903 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 03:06:25.228837 4011 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41338
I20260926 03:06:25.229966 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 4072
I20260926 03:06:25.242648 4474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:25.230980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51352 (local address 127.0.97.1:33279)
0926 03:06:25.231210 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:25.231216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:25.231385 (+ 169us) server_negotiation.cc:407] Connection header received
0926 03:06:25.231553 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:25.231558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:25.231630 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:25.231710 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:25.232915 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:25.233567 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:25.234874 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:25.235341 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:25.235455 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:25.237632 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:25.237636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:25.237640 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:25.237673 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:25.240194 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:25.240854 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:25.240859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:25.240866 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:25.240993 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:25.241322 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:25.241327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:25.241330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:25.241585 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:25.241805 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:25.242107 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 03:06:25.242309 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 03:06:25.243242 4195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:25.325938 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 4072
I20260926 03:06:25.338817 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 4206
Sep 26 03:06:25 dist-test-slave-hmbw krb5kdc[3966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:06:25.355661 4479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:25.339689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54142 (local address 127.0.97.2:42547)
0926 03:06:25.340300 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:25.340308 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:25.340334 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:06:25.340392 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:25.340399 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:25.340605 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:25.340740 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:25.341912 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:25.343090 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:25.344822 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:25.345436 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:25.345550 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:25.349881 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:25.349904 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:25.349920 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:25.349969 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:25.352827 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:25.353580 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:25.353588 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:25.353591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:25.353724 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:25.354086 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:25.354090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:25.354092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:25.354445 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:25.354781 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:25.355057 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 03:06:25.355312 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":310,"threads_started":1}
I20260926 03:06:25.356989 4329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:25.439097 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 4206
I20260926 03:06:25.451676 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 4339
Sep 26 03:06:25 dist-test-slave-hmbw krb5kdc[3966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:06:25.468772 4481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:25.452757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52336 (local address 127.0.97.3:45947)
0926 03:06:25.453463 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:25.453470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:25.453500 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:06:25.453565 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:25.453574 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:25.453808 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:25.453939 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:25.455365 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:25.456399 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:25.457716 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:25.458439 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:25.458583 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:25.463127 (+ 4544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:25.463142 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:25.463158 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:25.463208 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:25.465943 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:25.466745 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:25.466753 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:25.466755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:25.466874 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:25.467265 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:25.467269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:25.467271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:25.467660 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:25.467898 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:25.468221 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 03:06:25.468426 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":369,"threads_started":1}
I20260926 03:06:25.470098 4462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:25.552093 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 4339
I20260926 03:06:25.568176 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 3979
I20260926 03:06:25.569355 4041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:25.670867 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 3979
2026-09-26T03:06:25Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3879 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:06:25 dist-test-slave-hmbw krb5kdc[4486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:06:25 dist-test-slave-hmbw krb5kdc[4486](info): set up 2 sockets
Sep 26 03:06:25 dist-test-slave-hmbw krb5kdc[4486](info): commencing operation
krb5kdc: starting...
W20260926 03:06:26.679041 388 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:06:26 dist-test-slave-hmbw krb5kdc[4486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:06:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:06:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:06:26.875355 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40701
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:27.199608 4499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:27.199931 4499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:27.200065 4499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:27.207305 4499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:06:27.207423 4499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:27.207475 4499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:27.207549 4499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:06:27.207609 4499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:06:27.217568 4499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:27.218852 4499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:27.220144 4499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:27.231108 4504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:27.231247 4507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:27.231261 4505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:27.232177 4499 server_base.cc:1034] running on GCE node
I20260926 03:06:27.232952 4499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:27.234380 4499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:27.235602 4499 hybrid_clock.cc:648] HybridClock initialized: now 1790391987235555 us; error 41 us; skew 500 ppm
I20260926 03:06:27.235971 4499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:27 dist-test-slave-hmbw krb5kdc[4486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:27.239037 4499 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:06:27.240192 4499 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:27.240288 4499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:27.240574 4499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:27.243575 4499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "ed42afccda24431c90921d2a66d60e95"
format_stamp: "Formatted at 2026-09-26 03:06:27 on dist-test-slave-hmbw"
server_key: "a3f85f1b5192c05383955540dd5bcbb7"
server_key_iv: "94ba3468074d3b3f97a6447448988c4f"
server_key_version: "encryptionkey@0"
I20260926 03:06:27.244349 4499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "ed42afccda24431c90921d2a66d60e95"
format_stamp: "Formatted at 2026-09-26 03:06:27 on dist-test-slave-hmbw"
server_key: "a3f85f1b5192c05383955540dd5bcbb7"
server_key_iv: "94ba3468074d3b3f97a6447448988c4f"
server_key_version: "encryptionkey@0"
I20260926 03:06:27.249547 4499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:06:27.253382 4513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:27.254874 4499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:06:27.255054 4499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "ed42afccda24431c90921d2a66d60e95"
format_stamp: "Formatted at 2026-09-26 03:06:27 on dist-test-slave-hmbw"
server_key: "a3f85f1b5192c05383955540dd5bcbb7"
server_key_iv: "94ba3468074d3b3f97a6447448988c4f"
server_key_version: "encryptionkey@0"
I20260926 03:06:27.255288 4499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:27.280695 4499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:27.285269 4499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:27.285530 4499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:27.303148 4499 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40701
I20260926 03:06:27.303115 4564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40701 every 8 connection(s)
I20260926 03:06:27.304100 4499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:06:27.307889 4565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:27.313793 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 4499
I20260926 03:06:27.314013 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:06:27.314637 388 external_mini_cluster.cc:1428] Setting key 89d275317bb8ea79a9bf7f6af771e19d
I20260926 03:06:27.319052 4565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed42afccda24431c90921d2a66d60e95: Bootstrap starting.
I20260926 03:06:27.323033 4565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed42afccda24431c90921d2a66d60e95: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:27.324213 4565 log.cc:824] T 00000000000000000000000000000000 P ed42afccda24431c90921d2a66d60e95: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:27.327380 4565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed42afccda24431c90921d2a66d60e95: No bootstrap required, opened a new log
Sep 26 03:06:27 dist-test-slave-hmbw krb5kdc[4486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:27.335752 4565 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed42afccda24431c90921d2a66d60e95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed42afccda24431c90921d2a66d60e95" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40701 } }
I20260926 03:06:27.336120 4565 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed42afccda24431c90921d2a66d60e95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:27.336236 4565 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed42afccda24431c90921d2a66d60e95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed42afccda24431c90921d2a66d60e95, State: Initialized, Role: FOLLOWER
I20260926 03:06:27.336982 4565 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed42afccda24431c90921d2a66d60e95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed42afccda24431c90921d2a66d60e95" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40701 } }
I20260926 03:06:27.337332 4565 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed42afccda24431c90921d2a66d60e95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:06:27.337533 4565 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed42afccda24431c90921d2a66d60e95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:06:27.337733 4565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed42afccda24431c90921d2a66d60e95 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:27.338630 4568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:27.316770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48370 (local address 127.0.97.62:40701)
0926 03:06:27.317576 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:27.317590 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:27.317634 (+ 44us) server_negotiation.cc:407] Connection header received
0926 03:06:27.319459 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:27.319486 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:27.319920 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:27.320280 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:27.322217 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:27.323813 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:27.325501 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:27.326192 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:27.326379 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:27.330727 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:27.330736 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:27.330756 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:27.330799 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:27.334129 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:27.334995 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:27.335003 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:27.335011 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:27.335195 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:27.335585 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:27.335589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:27.335591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:27.336242 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:27.336498 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:27.337005 (+ 507us) server_negotiation.cc:299] Negotiation successful
0926 03:06:27.337429 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":347,"threads_started":1}
I20260926 03:06:27.340356 4565 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed42afccda24431c90921d2a66d60e95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed42afccda24431c90921d2a66d60e95" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40701 } }
I20260926 03:06:27.340978 4565 leader_election.cc:302] T 00000000000000000000000000000000 P ed42afccda24431c90921d2a66d60e95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed42afccda24431c90921d2a66d60e95; no voters:
I20260926 03:06:27.341660 4565 leader_election.cc:288] T 00000000000000000000000000000000 P ed42afccda24431c90921d2a66d60e95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:06:27.341861 4570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed42afccda24431c90921d2a66d60e95 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:27.342686 4570 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed42afccda24431c90921d2a66d60e95 [term 1 LEADER]: Becoming Leader. State: Replica: ed42afccda24431c90921d2a66d60e95, State: Running, Role: LEADER
I20260926 03:06:27.343277 4570 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed42afccda24431c90921d2a66d60e95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed42afccda24431c90921d2a66d60e95" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40701 } }
I20260926 03:06:27.343916 4565 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed42afccda24431c90921d2a66d60e95 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:06:27.345789 4572 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed42afccda24431c90921d2a66d60e95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed42afccda24431c90921d2a66d60e95. Latest consensus state: current_term: 1 leader_uuid: "ed42afccda24431c90921d2a66d60e95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed42afccda24431c90921d2a66d60e95" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40701 } } }
I20260926 03:06:27.346360 4572 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed42afccda24431c90921d2a66d60e95 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:27.347276 4571 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed42afccda24431c90921d2a66d60e95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed42afccda24431c90921d2a66d60e95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed42afccda24431c90921d2a66d60e95" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40701 } } }
I20260926 03:06:27.347767 4571 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed42afccda24431c90921d2a66d60e95 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:27.349699 4575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:06:27.355008 4575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:06:27.364017 4575 catalog_manager.cc:1347] Generated new cluster ID: 320def77bff548fc9655de2ba0d110db
I20260926 03:06:27.364147 4575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:06:27.383852 4575 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:06:27.385804 4575 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:06:27.398821 4575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed42afccda24431c90921d2a66d60e95: Generated new TSK 0
I20260926 03:06:27.399947 4575 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:06:27.454397 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40701
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:27.789549 4591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:27.789884 4591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:27.790155 4591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:27.797996 4591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:27.798161 4591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:27.798322 4591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:06:27.810333 4591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:27.811976 4591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:27.814188 4591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:27.831372 4596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:27.831400 4597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:27.831430 4599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:27.832067 4591 server_base.cc:1034] running on GCE node
I20260926 03:06:27.832680 4591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:27.833655 4591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:27.834892 4591 hybrid_clock.cc:648] HybridClock initialized: now 1790391987834839 us; error 42 us; skew 500 ppm
I20260926 03:06:27.835232 4591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:27 dist-test-slave-hmbw krb5kdc[4486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:27.838383 4591 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:06:27.839700 4591 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:27.839799 4591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:27.840063 4591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:27.843061 4591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-0/data/instance:
uuid: "818c46ba7ca14f2dbe41f76f0eeb11ae"
format_stamp: "Formatted at 2026-09-26 03:06:27 on dist-test-slave-hmbw"
server_key: "60f512ec3cbe4b047907c83e2f92a354"
server_key_iv: "1c11936d5db53a2122c485afe85812d3"
server_key_version: "encryptionkey@0"
I20260926 03:06:27.843907 4591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-0/wal/instance:
uuid: "818c46ba7ca14f2dbe41f76f0eeb11ae"
format_stamp: "Formatted at 2026-09-26 03:06:27 on dist-test-slave-hmbw"
server_key: "60f512ec3cbe4b047907c83e2f92a354"
server_key_iv: "1c11936d5db53a2122c485afe85812d3"
server_key_version: "encryptionkey@0"
I20260926 03:06:27.849416 4591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 03:06:27.853116 4605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:27.854617 4591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:06:27.854790 4591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "818c46ba7ca14f2dbe41f76f0eeb11ae"
format_stamp: "Formatted at 2026-09-26 03:06:27 on dist-test-slave-hmbw"
server_key: "60f512ec3cbe4b047907c83e2f92a354"
server_key_iv: "1c11936d5db53a2122c485afe85812d3"
server_key_version: "encryptionkey@0"
I20260926 03:06:27.855000 4591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:27.874085 4591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:27.878618 4591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:27.878880 4591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:27.879930 4591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:06:27.881368 4591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:27.881464 4591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:27.881570 4591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:27.881636 4591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:27.912324 4591 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41251
I20260926 03:06:27.912370 4717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41251 every 8 connection(s)
I20260926 03:06:27.913118 4591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:06:27.915726 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 4591
I20260926 03:06:27.915920 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:06:27.916705 388 external_mini_cluster.cc:1428] Setting key 4adf38c61694612e532de21405b8897e
Sep 26 03:06:27 dist-test-slave-hmbw krb5kdc[4486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:06:27.933265 4720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:27.915496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54355 (local address 127.0.97.62:40701)
0926 03:06:27.915928 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:27.915933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:27.917143 (+ 1210us) server_negotiation.cc:407] Connection header received
0926 03:06:27.919424 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:27.919429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:27.919510 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:27.919601 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:27.921192 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:27.921926 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:27.923596 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:27.923999 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:27.924110 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:27.927598 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:27.927604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:27.927607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:27.927651 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:27.930031 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:27.930881 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:27.930886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:27.930888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:27.931017 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:27.931434 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:27.931438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:27.931440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:27.931726 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:27.931859 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:27.932614 (+ 755us) server_negotiation.cc:299] Negotiation successful
0926 03:06:27.932852 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":233,"threads_started":1}
I20260926 03:06:27.934228 4721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:27.916165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40701 (local address 127.0.97.1:54355)
0926 03:06:27.916979 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:27.917018 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:27.919154 (+ 2136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:27.919863 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:27.919873 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:27.920421 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:27.921010 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:27.921028 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:27.922087 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:27.922091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:27.923433 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:27.923446 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:27.924236 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:27.924244 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:27.924460 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:27.925115 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:27.925147 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:27.927341 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:27.930204 (+ 2863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:27.930214 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:27.930233 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:27.930750 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:27.931122 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:27.931127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:27.931131 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:27.931318 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:27.931859 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:27.931868 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:27.932220 (+ 352us) client_negotiation.cc:769] Sending connection context
0926 03:06:27.932481 (+ 261us) client_negotiation.cc:240] Negotiation successful
0926 03:06:27.933043 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":342,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 03:06:27.936100 4718 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40701
I20260926 03:06:27.936462 4718 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:27.937263 4718 heartbeater.cc:511] Master 127.0.97.62:40701 requested a full tablet report, sending...
I20260926 03:06:27.939612 4530 ts_manager.cc:194] Registered new tserver with Master: 818c46ba7ca14f2dbe41f76f0eeb11ae (127.0.97.1:41251)
I20260926 03:06:27.941237 4530 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:06:27.959985 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40701
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:28.294463 4724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:28.294791 4724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:28.294888 4724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:28.302376 4724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:28.302539 4724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:28.302711 4724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:06:28.312950 4724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:28.314363 4724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:28.315747 4724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:28.329190 4729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:28.329219 4730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:28.329383 4732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:28.331436 4724 server_base.cc:1034] running on GCE node
I20260926 03:06:28.332079 4724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:28.333076 4724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:28.334295 4724 hybrid_clock.cc:648] HybridClock initialized: now 1790391988334232 us; error 55 us; skew 500 ppm
I20260926 03:06:28.334672 4724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:28 dist-test-slave-hmbw krb5kdc[4486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:28.338294 4724 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:06:28.339584 4724 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:28.339751 4724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:28.340075 4724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:28.343139 4724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-1/data/instance:
uuid: "7c16e68d7148491a9ebb496e863057e0"
format_stamp: "Formatted at 2026-09-26 03:06:28 on dist-test-slave-hmbw"
server_key: "0693b867f982c9c481ae779f39b30e69"
server_key_iv: "c9b743bb69a059dbd763affaa2d73da9"
server_key_version: "encryptionkey@0"
I20260926 03:06:28.343989 4724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7c16e68d7148491a9ebb496e863057e0"
format_stamp: "Formatted at 2026-09-26 03:06:28 on dist-test-slave-hmbw"
server_key: "0693b867f982c9c481ae779f39b30e69"
server_key_iv: "c9b743bb69a059dbd763affaa2d73da9"
server_key_version: "encryptionkey@0"
I20260926 03:06:28.349535 4724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 03:06:28.353644 4738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:28.355520 4724 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260926 03:06:28.355691 4724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "7c16e68d7148491a9ebb496e863057e0"
format_stamp: "Formatted at 2026-09-26 03:06:28 on dist-test-slave-hmbw"
server_key: "0693b867f982c9c481ae779f39b30e69"
server_key_iv: "c9b743bb69a059dbd763affaa2d73da9"
server_key_version: "encryptionkey@0"
I20260926 03:06:28.355934 4724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:28.378674 4724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:28.383754 4724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:28.384039 4724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:28.385092 4724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:06:28.386550 4724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:28.386652 4724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:28.386732 4724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:28.386802 4724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:28.419050 4724 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44711
I20260926 03:06:28.419106 4850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44711 every 8 connection(s)
I20260926 03:06:28.419867 4724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:06:28.429872 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 4724
I20260926 03:06:28.430048 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:06:28.430621 388 external_mini_cluster.cc:1428] Setting key 2cb9924dd3a8e3eeab845db513992443
Sep 26 03:06:28 dist-test-slave-hmbw krb5kdc[4486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:28.441236 4720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:28.422209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45257 (local address 127.0.97.62:40701)
0926 03:06:28.422398 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:28.422402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:28.423719 (+ 1317us) server_negotiation.cc:407] Connection header received
0926 03:06:28.425595 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:28.425600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:28.425677 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:28.425800 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:28.427625 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:28.428340 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:28.430135 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:28.430920 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:28.431069 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:28.434835 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:28.434841 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:28.434846 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:28.434886 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:28.437940 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:28.439004 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:28.439008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:28.439011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:28.439141 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:28.439589 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:28.439592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:28.439593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:28.439850 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:28.439951 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:28.440763 (+ 812us) server_negotiation.cc:299] Negotiation successful
0926 03:06:28.440991 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 03:06:28.442607 4853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:28.422713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40701 (local address 127.0.97.2:45257)
0926 03:06:28.423529 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:28.423575 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:28.425277 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:28.426135 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:28.426145 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:28.426769 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:28.427388 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:28.427409 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:28.428504 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:28.428508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:28.429926 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:28.429940 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:28.431173 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:28.431180 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:28.431494 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:28.432216 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:28.432239 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:28.434562 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:28.438122 (+ 3560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:28.438130 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:28.438181 (+ 51us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:28.438864 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:28.439249 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:28.439253 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:28.439257 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:28.439495 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:28.439963 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:28.439971 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:28.440405 (+ 434us) client_negotiation.cc:769] Sending connection context
0926 03:06:28.440687 (+ 282us) client_negotiation.cc:240] Negotiation successful
0926 03:06:28.441241 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":283,"threads_started":1}
I20260926 03:06:28.444187 4851 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40701
I20260926 03:06:28.444548 4851 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:28.445348 4851 heartbeater.cc:511] Master 127.0.97.62:40701 requested a full tablet report, sending...
I20260926 03:06:28.447258 4530 ts_manager.cc:194] Registered new tserver with Master: 7c16e68d7148491a9ebb496e863057e0 (127.0.97.2:44711)
I20260926 03:06:28.448202 4530 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45257
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:06:28.479198 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40701
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:28.796900 4856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:28.797205 4856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:28.797336 4856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:28.804600 4856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:28.804746 4856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:28.804899 4856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:06:28.814834 4856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40701
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:28.816177 4856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:28.817472 4856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:28.830003 4861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:28.829970 4862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:28.829938 4864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:28.831305 4856 server_base.cc:1034] running on GCE node
I20260926 03:06:28.831969 4856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:28.832930 4856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:28.834153 4856 hybrid_clock.cc:648] HybridClock initialized: now 1790391988834094 us; error 45 us; skew 500 ppm
I20260926 03:06:28.834519 4856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:28 dist-test-slave-hmbw krb5kdc[4486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:28.842021 4856 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:06:28.843314 4856 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:28.843420 4856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:28.843760 4856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:28.846921 4856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-2/data/instance:
uuid: "85f0212e632c495a804ff5a2854735df"
format_stamp: "Formatted at 2026-09-26 03:06:28 on dist-test-slave-hmbw"
server_key: "b0413c9ba58bc40583ac7a83048ccddd"
server_key_iv: "3a966a448877170fd63d662979bb6dd9"
server_key_version: "encryptionkey@0"
I20260926 03:06:28.847774 4856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-2/wal/instance:
uuid: "85f0212e632c495a804ff5a2854735df"
format_stamp: "Formatted at 2026-09-26 03:06:28 on dist-test-slave-hmbw"
server_key: "b0413c9ba58bc40583ac7a83048ccddd"
server_key_iv: "3a966a448877170fd63d662979bb6dd9"
server_key_version: "encryptionkey@0"
I20260926 03:06:28.853515 4856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:06:28.857535 4870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:28.859133 4856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 03:06:28.859301 4856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "85f0212e632c495a804ff5a2854735df"
format_stamp: "Formatted at 2026-09-26 03:06:28 on dist-test-slave-hmbw"
server_key: "b0413c9ba58bc40583ac7a83048ccddd"
server_key_iv: "3a966a448877170fd63d662979bb6dd9"
server_key_version: "encryptionkey@0"
I20260926 03:06:28.859508 4856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:28.883873 4856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:28.888182 4856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:28.888437 4856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:28.889452 4856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:06:28.891232 4856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:28.891340 4856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:28.891434 4856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:28.891541 4856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 03:06:28.921764 4856 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37629
I20260926 03:06:28.921851 4982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37629 every 8 connection(s)
I20260926 03:06:28.922571 4856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:06:28.928584 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 4856
I20260926 03:06:28.928783 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:06:28.929190 388 external_mini_cluster.cc:1428] Setting key 9a6b16b18fa1ee2fa98650a92ea6e7f7
Sep 26 03:06:28 dist-test-slave-hmbw krb5kdc[4486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:28.942715 4720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:28.925131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52701 (local address 127.0.97.62:40701)
0926 03:06:28.925315 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:28.925319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:28.926556 (+ 1237us) server_negotiation.cc:407] Connection header received
0926 03:06:28.928469 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:28.928476 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:28.928563 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:28.928651 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:28.930407 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:28.931400 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:28.933128 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:28.933529 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:28.933628 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:28.937275 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:28.937281 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:28.937285 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:28.937324 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:28.939809 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:28.940596 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:28.940600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:28.940602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:28.940718 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:28.941076 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:28.941079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:28.941082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:28.941356 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:28.941467 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:28.942219 (+ 752us) server_negotiation.cc:299] Negotiation successful
0926 03:06:28.942420 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:06:28.943991 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:28.925611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40701 (local address 127.0.97.3:52701)
0926 03:06:28.926354 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:28.926397 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:28.928129 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:28.928930 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:28.928939 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:28.929502 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:28.930192 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:28.930222 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:28.931574 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:28.931579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:28.932981 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:28.932995 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:28.933737 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:28.933742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:28.933936 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:28.934621 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:28.934648 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:28.937013 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:28.939988 (+ 2975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:28.939997 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:28.940012 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:28.940494 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:28.940805 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:28.940808 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:28.940810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:28.940985 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:28.941500 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:28.941507 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:28.941874 (+ 367us) client_negotiation.cc:769] Sending connection context
0926 03:06:28.942131 (+ 257us) client_negotiation.cc:240] Negotiation successful
0926 03:06:28.942630 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":286,"threads_started":1}
I20260926 03:06:28.944253 4718 heartbeater.cc:503] Master 127.0.97.62:40701 was elected leader, sending a full tablet report...
I20260926 03:06:28.945389 4983 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40701
I20260926 03:06:28.945706 4983 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:28.946346 4983 heartbeater.cc:511] Master 127.0.97.62:40701 requested a full tablet report, sending...
I20260926 03:06:28.947721 4529 ts_manager.cc:194] Registered new tserver with Master: 85f0212e632c495a804ff5a2854735df (127.0.97.3:37629)
I20260926 03:06:28.948550 4529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52701
I20260926 03:06:28.957701 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:06:28.974856 4720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:28.962382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48382 (local address 127.0.97.62:40701)
0926 03:06:28.962655 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:28.962663 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:28.963109 (+ 446us) server_negotiation.cc:407] Connection header received
0926 03:06:28.963249 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:28.963253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:28.963320 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:28.963405 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:28.964928 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:28.965651 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:28.966971 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:28.967416 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:28.967518 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:28.970014 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:28.970019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:28.970023 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:28.970056 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:28.972455 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:28.973103 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:28.973107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:28.973109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:28.973217 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:28.973508 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:28.973510 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:28.973512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:28.973831 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:28.974007 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:28.974322 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 03:06:28.974491 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 03:06:28.978828 4529 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:06:28.985389 4529 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:06:29.021474 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:28.998534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41251 (local address 127.0.0.1:33258)
0926 03:06:29.011758 (+ 13224us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:29.011781 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:29.011938 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:29.013085 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:29.013094 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:29.013140 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:29.013572 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:29.013585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:29.015564 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:29.015569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:29.018134 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:29.018152 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:29.020433 (+ 2281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:29.020440 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:29.020678 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:29.020750 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 03:06:29.020892 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 03:06:29.021091 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13051,"thread_start_us":11917,"threads_started":1}
I20260926 03:06:29.022467 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:28.999086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33258 (local address 127.0.97.1:41251)
0926 03:06:28.999755 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:28.999766 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:29.012089 (+ 12323us) server_negotiation.cc:407] Connection header received
0926 03:06:29.012217 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:29.012227 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:29.012529 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:29.012773 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:29.013796 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:29.015405 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:29.018326 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:29.019875 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:29.020017 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:29.020278 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:29.021963 (+ 1685us) server_negotiation.cc:299] Negotiation successful
0926 03:06:29.022155 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":272,"threads_started":1}
I20260926 03:06:29.023041 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:28.998211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37629 (local address 127.0.0.1:57652)
0926 03:06:29.009908 (+ 11697us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:29.009978 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:29.010206 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:29.015375 (+ 5169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:29.015379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:29.015407 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:29.015910 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:29.015924 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:29.018235 (+ 2311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:29.018240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:29.020367 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:29.020379 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:29.022310 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:29.022318 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:29.022487 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:29.022548 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 03:06:29.022615 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 03:06:29.022746 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11327,"thread_start_us":15680,"threads_started":1}
I20260926 03:06:29.024087 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:28.998458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57652 (local address 127.0.97.3:37629)
0926 03:06:29.014712 (+ 16254us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:29.014724 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:29.014761 (+ 37us) server_negotiation.cc:407] Connection header received
0926 03:06:29.014862 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:29.014870 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:29.015128 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:29.016192 (+ 1064us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:29.016485 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:29.018059 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:29.020606 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:29.022060 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:29.023312 (+ 1252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:29.023549 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:29.023623 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 03:06:29.023783 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15849,"thread_start_us":16020,"threads_started":1}
I20260926 03:06:29.026666 4653 tablet_service.cc:1467] Processing CreateTablet for tablet 569fe3847ac3446ea1d1d4a2b1550c21 (DEFAULT_TABLE table=test-table [id=1b1cf6410efa48b8bdb5c248a0a6b3b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:29.027191 4918 tablet_service.cc:1467] Processing CreateTablet for tablet 569fe3847ac3446ea1d1d4a2b1550c21 (DEFAULT_TABLE table=test-table [id=1b1cf6410efa48b8bdb5c248a0a6b3b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:29.028801 4918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 569fe3847ac3446ea1d1d4a2b1550c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:29.029780 4653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 569fe3847ac3446ea1d1d4a2b1550c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:29.033319 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:29.000871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41030 (local address 127.0.97.2:44711)
0926 03:06:29.020017 (+ 19146us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:29.020028 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:29.021022 (+ 994us) server_negotiation.cc:407] Connection header received
0926 03:06:29.021145 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:29.021155 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:29.021444 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:29.021945 (+ 501us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:29.022462 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:29.024000 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:29.026303 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:29.027756 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:29.029210 (+ 1454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:29.029429 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:29.032757 (+ 3328us) server_negotiation.cc:299] Negotiation successful
0926 03:06:29.032970 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18938,"thread_start_us":19161,"threads_started":1}
I20260926 03:06:29.033932 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:29.000323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44711 (local address 127.0.0.1:41030)
0926 03:06:29.020729 (+ 20406us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:29.020750 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:29.020890 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:29.021923 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:29.021927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:29.021955 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:29.022314 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:29.022324 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:29.024177 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:29.024182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:29.026097 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:29.026122 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:29.029590 (+ 3468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:29.029598 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:29.030023 (+ 425us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:29.030061 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 03:06:29.030166 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 03:06:29.030320 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20277,"thread_start_us":21315,"threads_started":1}
I20260926 03:06:29.041496 4786 tablet_service.cc:1467] Processing CreateTablet for tablet 569fe3847ac3446ea1d1d4a2b1550c21 (DEFAULT_TABLE table=test-table [id=1b1cf6410efa48b8bdb5c248a0a6b3b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:29.043035 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 569fe3847ac3446ea1d1d4a2b1550c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:29.046029 5001 tablet_bootstrap.cc:492] T 569fe3847ac3446ea1d1d4a2b1550c21 P 818c46ba7ca14f2dbe41f76f0eeb11ae: Bootstrap starting.
I20260926 03:06:29.050470 5001 tablet_bootstrap.cc:654] T 569fe3847ac3446ea1d1d4a2b1550c21 P 818c46ba7ca14f2dbe41f76f0eeb11ae: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:29.051928 5001 log.cc:824] T 569fe3847ac3446ea1d1d4a2b1550c21 P 818c46ba7ca14f2dbe41f76f0eeb11ae: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:29.055106 5003 tablet_bootstrap.cc:492] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df: Bootstrap starting.
I20260926 03:06:29.055320 5001 tablet_bootstrap.cc:492] T 569fe3847ac3446ea1d1d4a2b1550c21 P 818c46ba7ca14f2dbe41f76f0eeb11ae: No bootstrap required, opened a new log
I20260926 03:06:29.055603 5001 ts_tablet_manager.cc:1397] T 569fe3847ac3446ea1d1d4a2b1550c21 P 818c46ba7ca14f2dbe41f76f0eeb11ae: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260926 03:06:29.058147 5003 tablet_bootstrap.cc:654] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:29.059643 5003 log.cc:824] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:29.063921 5003 tablet_bootstrap.cc:492] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df: No bootstrap required, opened a new log
I20260926 03:06:29.064304 5003 ts_tablet_manager.cc:1397] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260926 03:06:29.066524 5001 raft_consensus.cc:348] T 569fe3847ac3446ea1d1d4a2b1550c21 P 818c46ba7ca14f2dbe41f76f0eeb11ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f0212e632c495a804ff5a2854735df" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37629 } } peers { permanent_uuid: "818c46ba7ca14f2dbe41f76f0eeb11ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41251 } } peers { permanent_uuid: "7c16e68d7148491a9ebb496e863057e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44711 } }
I20260926 03:06:29.067016 5001 raft_consensus.cc:374] T 569fe3847ac3446ea1d1d4a2b1550c21 P 818c46ba7ca14f2dbe41f76f0eeb11ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:29.067113 5001 raft_consensus.cc:729] T 569fe3847ac3446ea1d1d4a2b1550c21 P 818c46ba7ca14f2dbe41f76f0eeb11ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818c46ba7ca14f2dbe41f76f0eeb11ae, State: Initialized, Role: FOLLOWER
I20260926 03:06:29.067164 5005 tablet_bootstrap.cc:492] T 569fe3847ac3446ea1d1d4a2b1550c21 P 7c16e68d7148491a9ebb496e863057e0: Bootstrap starting.
I20260926 03:06:29.067755 5001 consensus_queue.cc:260] T 569fe3847ac3446ea1d1d4a2b1550c21 P 818c46ba7ca14f2dbe41f76f0eeb11ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f0212e632c495a804ff5a2854735df" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37629 } } peers { permanent_uuid: "818c46ba7ca14f2dbe41f76f0eeb11ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41251 } } peers { permanent_uuid: "7c16e68d7148491a9ebb496e863057e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44711 } }
I20260926 03:06:29.069484 5001 ts_tablet_manager.cc:1428] T 569fe3847ac3446ea1d1d4a2b1550c21 P 818c46ba7ca14f2dbe41f76f0eeb11ae: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260926 03:06:29.070880 5005 tablet_bootstrap.cc:654] T 569fe3847ac3446ea1d1d4a2b1550c21 P 7c16e68d7148491a9ebb496e863057e0: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:29.072257 5005 log.cc:824] T 569fe3847ac3446ea1d1d4a2b1550c21 P 7c16e68d7148491a9ebb496e863057e0: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:29.076339 5003 raft_consensus.cc:348] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f0212e632c495a804ff5a2854735df" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37629 } } peers { permanent_uuid: "818c46ba7ca14f2dbe41f76f0eeb11ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41251 } } peers { permanent_uuid: "7c16e68d7148491a9ebb496e863057e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44711 } }
I20260926 03:06:29.076929 5003 raft_consensus.cc:374] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:29.077030 5003 raft_consensus.cc:729] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f0212e632c495a804ff5a2854735df, State: Initialized, Role: FOLLOWER
I20260926 03:06:29.077912 5003 consensus_queue.cc:260] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f0212e632c495a804ff5a2854735df" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37629 } } peers { permanent_uuid: "818c46ba7ca14f2dbe41f76f0eeb11ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41251 } } peers { permanent_uuid: "7c16e68d7148491a9ebb496e863057e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44711 } }
I20260926 03:06:29.079246 4983 heartbeater.cc:503] Master 127.0.97.62:40701 was elected leader, sending a full tablet report...
I20260926 03:06:29.081596 5003 ts_tablet_manager.cc:1428] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df: Time spent starting tablet: real 0.017s user 0.013s sys 0.001s
I20260926 03:06:29.084116 5005 tablet_bootstrap.cc:492] T 569fe3847ac3446ea1d1d4a2b1550c21 P 7c16e68d7148491a9ebb496e863057e0: No bootstrap required, opened a new log
I20260926 03:06:29.084517 5005 ts_tablet_manager.cc:1397] T 569fe3847ac3446ea1d1d4a2b1550c21 P 7c16e68d7148491a9ebb496e863057e0: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.004s
I20260926 03:06:29.091862 5005 raft_consensus.cc:348] T 569fe3847ac3446ea1d1d4a2b1550c21 P 7c16e68d7148491a9ebb496e863057e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f0212e632c495a804ff5a2854735df" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37629 } } peers { permanent_uuid: "818c46ba7ca14f2dbe41f76f0eeb11ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41251 } } peers { permanent_uuid: "7c16e68d7148491a9ebb496e863057e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44711 } }
I20260926 03:06:29.092250 5005 raft_consensus.cc:374] T 569fe3847ac3446ea1d1d4a2b1550c21 P 7c16e68d7148491a9ebb496e863057e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:29.092361 5005 raft_consensus.cc:729] T 569fe3847ac3446ea1d1d4a2b1550c21 P 7c16e68d7148491a9ebb496e863057e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c16e68d7148491a9ebb496e863057e0, State: Initialized, Role: FOLLOWER
I20260926 03:06:29.092927 5005 consensus_queue.cc:260] T 569fe3847ac3446ea1d1d4a2b1550c21 P 7c16e68d7148491a9ebb496e863057e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f0212e632c495a804ff5a2854735df" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37629 } } peers { permanent_uuid: "818c46ba7ca14f2dbe41f76f0eeb11ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41251 } } peers { permanent_uuid: "7c16e68d7148491a9ebb496e863057e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44711 } }
I20260926 03:06:29.093976 4851 heartbeater.cc:503] Master 127.0.97.62:40701 was elected leader, sending a full tablet report...
I20260926 03:06:29.094651 5005 ts_tablet_manager.cc:1428] T 569fe3847ac3446ea1d1d4a2b1550c21 P 7c16e68d7148491a9ebb496e863057e0: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
W20260926 03:06:29.165385 4719 tablet.cc:2369] T 569fe3847ac3446ea1d1d4a2b1550c21 P 818c46ba7ca14f2dbe41f76f0eeb11ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:06:29.171774 4852 tablet.cc:2369] T 569fe3847ac3446ea1d1d4a2b1550c21 P 7c16e68d7148491a9ebb496e863057e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:06:29.173924 4984 tablet.cc:2369] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:06:29.184266 5008 raft_consensus.cc:482] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:06:29.184562 5008 raft_consensus.cc:504] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f0212e632c495a804ff5a2854735df" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37629 } } peers { permanent_uuid: "818c46ba7ca14f2dbe41f76f0eeb11ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41251 } } peers { permanent_uuid: "7c16e68d7148491a9ebb496e863057e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44711 } }
I20260926 03:06:29.186194 5008 leader_election.cc:288] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818c46ba7ca14f2dbe41f76f0eeb11ae (127.0.97.1:41251), 7c16e68d7148491a9ebb496e863057e0 (127.0.97.2:44711)
I20260926 03:06:29.192772 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:29.186531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41251 (local address 127.0.97.3:44445)
0926 03:06:29.186923 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:29.186942 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:29.187077 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:29.187442 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:29.187447 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:29.187484 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:29.187719 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:29.187728 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:29.188962 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:29.188967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:29.190845 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:29.190860 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:29.192200 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:29.192205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:29.192333 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:29.192358 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 03:06:29.192426 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 03:06:29.192520 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"spinlock_wait_cycles":22784}
I20260926 03:06:29.192842 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:29.186798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44445 (local address 127.0.97.1:41251)
0926 03:06:29.186987 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:29.186992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:29.187015 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:06:29.187225 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:29.187230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:29.187307 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:29.187399 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:29.187864 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:29.188809 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:29.191000 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:29.192043 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:29.192180 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:29.192280 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:29.192468 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 03:06:29.192598 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:06:29.193506 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:29.186632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36049 (local address 127.0.97.2:44711)
0926 03:06:29.186925 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:29.186931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:29.187478 (+ 547us) server_negotiation.cc:407] Connection header received
0926 03:06:29.187732 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:29.187738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:29.187817 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:29.187920 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:29.188402 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:29.189276 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:29.191465 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:29.192576 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:29.192705 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:29.192791 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:29.193135 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 03:06:29.193270 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260926 03:06:29.193531 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:29.186683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44711 (local address 127.0.97.3:36049)
0926 03:06:29.187376 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:29.187404 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:29.187571 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:29.187955 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:29.187958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:29.187979 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:29.188254 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:29.188261 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:29.189466 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:29.189474 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:29.191326 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:29.191336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:29.192831 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:29.192837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:29.192995 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:29.193027 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 03:06:29.193101 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 03:06:29.193221 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":423,"threads_started":1}
I20260926 03:06:29.193775 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "569fe3847ac3446ea1d1d4a2b1550c21" candidate_uuid: "85f0212e632c495a804ff5a2854735df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818c46ba7ca14f2dbe41f76f0eeb11ae" is_pre_election: true
I20260926 03:06:29.194252 4673 raft_consensus.cc:2454] T 569fe3847ac3446ea1d1d4a2b1550c21 P 818c46ba7ca14f2dbe41f76f0eeb11ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f0212e632c495a804ff5a2854735df in term 0.
I20260926 03:06:29.194572 4806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "569fe3847ac3446ea1d1d4a2b1550c21" candidate_uuid: "85f0212e632c495a804ff5a2854735df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c16e68d7148491a9ebb496e863057e0" is_pre_election: true
I20260926 03:06:29.194929 4873 leader_election.cc:302] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818c46ba7ca14f2dbe41f76f0eeb11ae, 85f0212e632c495a804ff5a2854735df; no voters:
I20260926 03:06:29.195093 4806 raft_consensus.cc:2454] T 569fe3847ac3446ea1d1d4a2b1550c21 P 7c16e68d7148491a9ebb496e863057e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f0212e632c495a804ff5a2854735df in term 0.
I20260926 03:06:29.195351 5008 raft_consensus.cc:2781] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:06:29.195477 5008 raft_consensus.cc:482] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:06:29.195577 5008 raft_consensus.cc:3037] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:29.197765 5008 raft_consensus.cc:504] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f0212e632c495a804ff5a2854735df" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37629 } } peers { permanent_uuid: "818c46ba7ca14f2dbe41f76f0eeb11ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41251 } } peers { permanent_uuid: "7c16e68d7148491a9ebb496e863057e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44711 } }
I20260926 03:06:29.198338 5008 leader_election.cc:288] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df [CANDIDATE]: Term 1 election: Requested vote from peers 818c46ba7ca14f2dbe41f76f0eeb11ae (127.0.97.1:41251), 7c16e68d7148491a9ebb496e863057e0 (127.0.97.2:44711)
I20260926 03:06:29.198814 4806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "569fe3847ac3446ea1d1d4a2b1550c21" candidate_uuid: "85f0212e632c495a804ff5a2854735df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c16e68d7148491a9ebb496e863057e0"
I20260926 03:06:29.198874 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "569fe3847ac3446ea1d1d4a2b1550c21" candidate_uuid: "85f0212e632c495a804ff5a2854735df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818c46ba7ca14f2dbe41f76f0eeb11ae"
I20260926 03:06:29.199013 4806 raft_consensus.cc:3037] T 569fe3847ac3446ea1d1d4a2b1550c21 P 7c16e68d7148491a9ebb496e863057e0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:29.199060 4673 raft_consensus.cc:3037] T 569fe3847ac3446ea1d1d4a2b1550c21 P 818c46ba7ca14f2dbe41f76f0eeb11ae [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:29.202055 4673 raft_consensus.cc:2454] T 569fe3847ac3446ea1d1d4a2b1550c21 P 818c46ba7ca14f2dbe41f76f0eeb11ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f0212e632c495a804ff5a2854735df in term 1.
I20260926 03:06:29.202055 4806 raft_consensus.cc:2454] T 569fe3847ac3446ea1d1d4a2b1550c21 P 7c16e68d7148491a9ebb496e863057e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f0212e632c495a804ff5a2854735df in term 1.
I20260926 03:06:29.202520 4872 leader_election.cc:302] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7c16e68d7148491a9ebb496e863057e0, 818c46ba7ca14f2dbe41f76f0eeb11ae, 85f0212e632c495a804ff5a2854735df; no voters:
I20260926 03:06:29.202842 5008 raft_consensus.cc:2781] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:29.203271 5008 raft_consensus.cc:686] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df [term 1 LEADER]: Becoming Leader. State: Replica: 85f0212e632c495a804ff5a2854735df, State: Running, Role: LEADER
I20260926 03:06:29.203783 5008 consensus_queue.cc:237] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f0212e632c495a804ff5a2854735df" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37629 } } peers { permanent_uuid: "818c46ba7ca14f2dbe41f76f0eeb11ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41251 } } peers { permanent_uuid: "7c16e68d7148491a9ebb496e863057e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44711 } }
I20260926 03:06:29.208694 4530 catalog_manager.cc:5495] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df reported cstate change: term changed from 0 to 1, leader changed from <none> to 85f0212e632c495a804ff5a2854735df (127.0.97.3). New cstate: current_term: 1 leader_uuid: "85f0212e632c495a804ff5a2854735df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f0212e632c495a804ff5a2854735df" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818c46ba7ca14f2dbe41f76f0eeb11ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c16e68d7148491a9ebb496e863057e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44711 } health_report { overall_health: UNKNOWN } } }
I20260926 03:06:29.249303 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:29.243707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57660 (local address 127.0.97.3:37629)
0926 03:06:29.244004 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:29.244010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:29.244099 (+ 89us) server_negotiation.cc:407] Connection header received
0926 03:06:29.244354 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:29.244358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:29.244475 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:29.244606 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:06:29.245044 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:29.245814 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:29.247152 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:29.247545 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:29.247652 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:29.248126 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:06:29.248742 (+ 616us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:06:29.248879 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:29.248968 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 03:06:29.249051 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260926 03:06:29.257983 4806 raft_consensus.cc:1261] T 569fe3847ac3446ea1d1d4a2b1550c21 P 7c16e68d7148491a9ebb496e863057e0 [term 1 FOLLOWER]: Refusing update from remote peer 85f0212e632c495a804ff5a2854735df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:29.258009 4673 raft_consensus.cc:1261] T 569fe3847ac3446ea1d1d4a2b1550c21 P 818c46ba7ca14f2dbe41f76f0eeb11ae [term 1 FOLLOWER]: Refusing update from remote peer 85f0212e632c495a804ff5a2854735df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:29.259128 5008 consensus_queue.cc:1035] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c16e68d7148491a9ebb496e863057e0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:29.259419 5011 consensus_queue.cc:1035] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df [LEADER]: Connected to new peer: Peer: permanent_uuid: "818c46ba7ca14f2dbe41f76f0eeb11ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:29.284410 4529 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:06:29.284719 4529 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:06:29.289004 4529 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed42afccda24431c90921d2a66d60e95: Sending DeleteTablet for 3 replicas of tablet 569fe3847ac3446ea1d1d4a2b1550c21
I20260926 03:06:29.290216 4918 tablet_service.cc:1514] Processing DeleteTablet for tablet 569fe3847ac3446ea1d1d4a2b1550c21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57652
I20260926 03:06:29.290284 4653 tablet_service.cc:1514] Processing DeleteTablet for tablet 569fe3847ac3446ea1d1d4a2b1550c21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33258
I20260926 03:06:29.291806 4786 tablet_service.cc:1514] Processing DeleteTablet for tablet 569fe3847ac3446ea1d1d4a2b1550c21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41030
I20260926 03:06:29.293464 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 4591
I20260926 03:06:29.293491 5024 tablet_replica.cc:330] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df: stopping tablet replica
I20260926 03:06:29.294108 5024 raft_consensus.cc:2229] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:06:29.294836 5024 raft_consensus.cc:2258] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:29.297264 5024 ts_tablet_manager.cc:1905] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:29.302218 5025 tablet_replica.cc:330] T 569fe3847ac3446ea1d1d4a2b1550c21 P 7c16e68d7148491a9ebb496e863057e0: stopping tablet replica
I20260926 03:06:29.304791 5025 raft_consensus.cc:2229] T 569fe3847ac3446ea1d1d4a2b1550c21 P 7c16e68d7148491a9ebb496e863057e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:06:29.305097 5024 ts_tablet_manager.cc:1918] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:06:29.305270 5024 log.cc:1196] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-2/wal/wals/569fe3847ac3446ea1d1d4a2b1550c21
I20260926 03:06:29.305408 5025 raft_consensus.cc:2258] T 569fe3847ac3446ea1d1d4a2b1550c21 P 7c16e68d7148491a9ebb496e863057e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:29.305887 5024 ts_tablet_manager.cc:1939] T 569fe3847ac3446ea1d1d4a2b1550c21 P 85f0212e632c495a804ff5a2854735df: Deleting consensus metadata
I20260926 03:06:29.306769 5023 tablet_replica.cc:330] T 569fe3847ac3446ea1d1d4a2b1550c21 P 818c46ba7ca14f2dbe41f76f0eeb11ae: stopping tablet replica
I20260926 03:06:29.307255 5023 raft_consensus.cc:2229] T 569fe3847ac3446ea1d1d4a2b1550c21 P 818c46ba7ca14f2dbe41f76f0eeb11ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:06:29.307737 5023 raft_consensus.cc:2258] T 569fe3847ac3446ea1d1d4a2b1550c21 P 818c46ba7ca14f2dbe41f76f0eeb11ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:29.307827 4517 catalog_manager.cc:4841] TS 85f0212e632c495a804ff5a2854735df (127.0.97.3:37629): tablet 569fe3847ac3446ea1d1d4a2b1550c21 (table test-table [id=1b1cf6410efa48b8bdb5c248a0a6b3b7]) successfully deleted
I20260926 03:06:29.309892 5023 ts_tablet_manager.cc:1905] T 569fe3847ac3446ea1d1d4a2b1550c21 P 818c46ba7ca14f2dbe41f76f0eeb11ae: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 03:06:29 dist-test-slave-hmbw krb5kdc[4486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:06:29.317121 5023 ts_tablet_manager.cc:1918] T 569fe3847ac3446ea1d1d4a2b1550c21 P 818c46ba7ca14f2dbe41f76f0eeb11ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:06:29.317315 5023 log.cc:1196] T 569fe3847ac3446ea1d1d4a2b1550c21 P 818c46ba7ca14f2dbe41f76f0eeb11ae: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-0/wal/wals/569fe3847ac3446ea1d1d4a2b1550c21
I20260926 03:06:29.317804 5023 ts_tablet_manager.cc:1939] T 569fe3847ac3446ea1d1d4a2b1550c21 P 818c46ba7ca14f2dbe41f76f0eeb11ae: Deleting consensus metadata
I20260926 03:06:29.318120 5025 ts_tablet_manager.cc:1905] T 569fe3847ac3446ea1d1d4a2b1550c21 P 7c16e68d7148491a9ebb496e863057e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:29.319725 4516 catalog_manager.cc:4841] TS 818c46ba7ca14f2dbe41f76f0eeb11ae (127.0.97.1:41251): tablet 569fe3847ac3446ea1d1d4a2b1550c21 (table test-table [id=1b1cf6410efa48b8bdb5c248a0a6b3b7]) successfully deleted
I20260926 03:06:29.323922 5025 ts_tablet_manager.cc:1918] T 569fe3847ac3446ea1d1d4a2b1550c21 P 7c16e68d7148491a9ebb496e863057e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:06:29.324108 5025 log.cc:1196] T 569fe3847ac3446ea1d1d4a2b1550c21 P 7c16e68d7148491a9ebb496e863057e0: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391950058021-388-0/minicluster-data/ts-1/wal/wals/569fe3847ac3446ea1d1d4a2b1550c21
I20260926 03:06:29.324975 5025 ts_tablet_manager.cc:1939] T 569fe3847ac3446ea1d1d4a2b1550c21 P 7c16e68d7148491a9ebb496e863057e0: Deleting consensus metadata
I20260926 03:06:29.326017 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:29.294779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33262 (local address 127.0.97.1:41251)
0926 03:06:29.294929 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:29.294935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:29.294953 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:06:29.295228 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:29.295233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:29.295331 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:29.295403 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:29.297289 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:29.298592 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:29.300677 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:29.301449 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:29.301540 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:29.311945 (+ 10405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:29.311966 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:29.311986 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:29.312042 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:29.315922 (+ 3880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:29.322750 (+ 6828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:29.322762 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:29.322766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:29.322983 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:29.323603 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:29.323608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:29.323611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:29.324133 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:29.324472 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:29.325296 (+ 824us) server_negotiation.cc:299] Negotiation successful
0926 03:06:29.325596 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 03:06:29.326980 4515 catalog_manager.cc:4841] TS 7c16e68d7148491a9ebb496e863057e0 (127.0.97.2:44711): tablet 569fe3847ac3446ea1d1d4a2b1550c21 (table test-table [id=1b1cf6410efa48b8bdb5c248a0a6b3b7]) successfully deleted
I20260926 03:06:29.327682 4713 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:29.415452 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 4591
I20260926 03:06:29.436702 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 4724
Sep 26 03:06:29 dist-test-slave-hmbw krb5kdc[4486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:06:29.454164 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:29.437572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41044 (local address 127.0.97.2:44711)
0926 03:06:29.437737 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:29.437742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:29.437760 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:06:29.437930 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:29.437934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:29.438008 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:29.438093 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:29.439656 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:29.440402 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:29.442226 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:29.442856 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:29.442968 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:29.448199 (+ 5231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:29.448216 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:29.448232 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:29.448280 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:29.451290 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:29.452222 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:29.452231 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:29.452234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:29.452386 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:29.452782 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:29.452787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:29.452789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:29.453193 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:29.453405 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:29.453702 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 03:06:29.453873 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 03:06:29.454873 4846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:29.541651 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 4724
I20260926 03:06:29.555229 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 4856
Sep 26 03:06:29 dist-test-slave-hmbw krb5kdc[4486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:06:29.573462 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:29.556186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57664 (local address 127.0.97.3:37629)
0926 03:06:29.556373 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:29.556379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:29.556398 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:06:29.556451 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:29.556455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:29.556520 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:29.556619 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:29.557920 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:29.558831 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:29.561091 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:29.561602 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:29.561773 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:29.567455 (+ 5682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:29.567468 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:29.567483 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:29.567527 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:29.570489 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:29.571318 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:29.571328 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:29.571332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:29.571503 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:29.571897 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:29.571902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:29.571904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:29.572297 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:29.572588 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:29.572868 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 03:06:29.573085 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:06:29.574266 4978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:29.692770 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 4856
I20260926 03:06:29.708060 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 4499
I20260926 03:06:29.709151 4560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:29.814170 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 4499
2026-09-26T03:06:29Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4146 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T03:06:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:06:29Z Disabled control of system clock
I20260926 03:06:29.879053 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42805
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 03:06:30.200313 5034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:30.200618 5034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:30.200821 5034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:30.208236 5034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:06:30.208365 5034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:30.208402 5034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:30.208477 5034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:06:30.208534 5034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:06:30.218319 5034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42805
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:30.219619 5034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:30.221426 5034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:30.233204 5039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:30.233161 5040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:30.233183 5042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:30.234009 5034 server_base.cc:1034] running on GCE node
I20260926 03:06:30.234848 5034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:30.236254 5034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:30.237484 5034 hybrid_clock.cc:648] HybridClock initialized: now 1790391990237438 us; error 33 us; skew 500 ppm
I20260926 03:06:30.237867 5034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:06:30.240321 5034 webserver.cc:460] Webserver started at http://127.0.97.62:33831/ using document root <none> and password file <none>
I20260926 03:06:30.241030 5034 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:30.241137 5034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:30.241461 5034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:30.244549 5034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "18d2ee59b60e483fad71ae73925fc976"
format_stamp: "Formatted at 2026-09-26 03:06:30 on dist-test-slave-hmbw"
server_key: "d6c62235e95295e59f1506276daf929d"
server_key_iv: "27549f99489e16610c15fcd217728d57"
server_key_version: "encryptionkey@0"
I20260926 03:06:30.245428 5034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "18d2ee59b60e483fad71ae73925fc976"
format_stamp: "Formatted at 2026-09-26 03:06:30 on dist-test-slave-hmbw"
server_key: "d6c62235e95295e59f1506276daf929d"
server_key_iv: "27549f99489e16610c15fcd217728d57"
server_key_version: "encryptionkey@0"
I20260926 03:06:30.250980 5034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 03:06:30.254716 5048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:30.256143 5034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:06:30.256311 5034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "18d2ee59b60e483fad71ae73925fc976"
format_stamp: "Formatted at 2026-09-26 03:06:30 on dist-test-slave-hmbw"
server_key: "d6c62235e95295e59f1506276daf929d"
server_key_iv: "27549f99489e16610c15fcd217728d57"
server_key_version: "encryptionkey@0"
I20260926 03:06:30.256516 5034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:30.262368 5034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:30.263236 5034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:30.263517 5034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:30.281207 5034 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42805
I20260926 03:06:30.281240 5099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42805 every 8 connection(s)
I20260926 03:06:30.282547 5034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:06:30.286237 5100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:30.286586 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 5034
I20260926 03:06:30.286816 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:06:30.287411 388 external_mini_cluster.cc:1428] Setting key fcec081fc378bfcfb53f2c0d4785b8b7
I20260926 03:06:30.295728 5103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:30.289387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51736 (local address 127.0.97.62:42805)
0926 03:06:30.290451 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:30.290465 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:30.290531 (+ 66us) server_negotiation.cc:407] Connection header received
0926 03:06:30.292153 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:30.292177 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:30.292641 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:30.293022 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:30.294057 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:30.294063 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:30.294076 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:06:30.294092 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:30.294161 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:06:30.294177 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:30.294377 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:30.294545 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 03:06:30.294792 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":396,"threads_started":1}
I20260926 03:06:30.298879 5100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18d2ee59b60e483fad71ae73925fc976: Bootstrap starting.
I20260926 03:06:30.301864 5100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18d2ee59b60e483fad71ae73925fc976: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:30.303072 5100 log.cc:824] T 00000000000000000000000000000000 P 18d2ee59b60e483fad71ae73925fc976: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:30.305925 5100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18d2ee59b60e483fad71ae73925fc976: No bootstrap required, opened a new log
I20260926 03:06:30.313097 5100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18d2ee59b60e483fad71ae73925fc976 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d2ee59b60e483fad71ae73925fc976" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42805 } }
I20260926 03:06:30.313489 5100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18d2ee59b60e483fad71ae73925fc976 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:30.313580 5100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18d2ee59b60e483fad71ae73925fc976 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d2ee59b60e483fad71ae73925fc976, State: Initialized, Role: FOLLOWER
I20260926 03:06:30.314265 5100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18d2ee59b60e483fad71ae73925fc976 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d2ee59b60e483fad71ae73925fc976" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42805 } }
I20260926 03:06:30.314528 5100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18d2ee59b60e483fad71ae73925fc976 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:06:30.314653 5100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18d2ee59b60e483fad71ae73925fc976 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:06:30.314818 5100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18d2ee59b60e483fad71ae73925fc976 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:30.317013 5100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18d2ee59b60e483fad71ae73925fc976 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d2ee59b60e483fad71ae73925fc976" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42805 } }
I20260926 03:06:30.317493 5100 leader_election.cc:302] T 00000000000000000000000000000000 P 18d2ee59b60e483fad71ae73925fc976 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18d2ee59b60e483fad71ae73925fc976; no voters:
I20260926 03:06:30.318071 5100 leader_election.cc:288] T 00000000000000000000000000000000 P 18d2ee59b60e483fad71ae73925fc976 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:06:30.318238 5105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18d2ee59b60e483fad71ae73925fc976 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:30.318931 5105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18d2ee59b60e483fad71ae73925fc976 [term 1 LEADER]: Becoming Leader. State: Replica: 18d2ee59b60e483fad71ae73925fc976, State: Running, Role: LEADER
I20260926 03:06:30.319482 5105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18d2ee59b60e483fad71ae73925fc976 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d2ee59b60e483fad71ae73925fc976" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42805 } }
I20260926 03:06:30.320092 5100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18d2ee59b60e483fad71ae73925fc976 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:06:30.322054 5107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18d2ee59b60e483fad71ae73925fc976 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18d2ee59b60e483fad71ae73925fc976. Latest consensus state: current_term: 1 leader_uuid: "18d2ee59b60e483fad71ae73925fc976" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d2ee59b60e483fad71ae73925fc976" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42805 } } }
I20260926 03:06:30.322190 5106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18d2ee59b60e483fad71ae73925fc976 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18d2ee59b60e483fad71ae73925fc976" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d2ee59b60e483fad71ae73925fc976" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42805 } } }
I20260926 03:06:30.322433 5107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18d2ee59b60e483fad71ae73925fc976 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:30.322456 5106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18d2ee59b60e483fad71ae73925fc976 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:30.323468 5112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:06:30.329562 5112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:06:30.337664 5112 catalog_manager.cc:1347] Generated new cluster ID: 5457183d65f34fafbce4811ea1690e00
I20260926 03:06:30.337805 5112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:06:30.357481 5112 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:06:30.357967 5112 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:06:30.364498 5112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18d2ee59b60e483fad71ae73925fc976: Generated new TSK 0
I20260926 03:06:30.365506 5112 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:06:30.382150 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42805
--builtin_ntp_servers=127.0.97.20:46091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 03:06:30.711366 5124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:30.711663 5124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:30.711748 5124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:30.719439 5124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:30.719558 5124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:30.719717 5124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:06:30.729521 5124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:30.730906 5124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:30.732722 5124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:30.745911 5130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:30.745901 5132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:30.746034 5129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:30.746821 5124 server_base.cc:1034] running on GCE node
I20260926 03:06:30.747437 5124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:30.748414 5124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:30.749635 5124 hybrid_clock.cc:648] HybridClock initialized: now 1790391990749571 us; error 58 us; skew 500 ppm
I20260926 03:06:30.749967 5124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:06:30.752482 5124 webserver.cc:460] Webserver started at http://127.0.97.1:41391/ using document root <none> and password file <none>
I20260926 03:06:30.753175 5124 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:30.753262 5124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:30.753525 5124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:30.756527 5124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-0/data/instance:
uuid: "1590e75dbe034536987f9f2e2c70e453"
format_stamp: "Formatted at 2026-09-26 03:06:30 on dist-test-slave-hmbw"
server_key: "99eb5f589a0e28e4db1e23baba541578"
server_key_iv: "8d2f10d7796349ee1b4c2aafa83ea45d"
server_key_version: "encryptionkey@0"
I20260926 03:06:30.757355 5124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1590e75dbe034536987f9f2e2c70e453"
format_stamp: "Formatted at 2026-09-26 03:06:30 on dist-test-slave-hmbw"
server_key: "99eb5f589a0e28e4db1e23baba541578"
server_key_iv: "8d2f10d7796349ee1b4c2aafa83ea45d"
server_key_version: "encryptionkey@0"
I20260926 03:06:30.763068 5124 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 03:06:30.766871 5138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:30.768493 5124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:06:30.768693 5124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "1590e75dbe034536987f9f2e2c70e453"
format_stamp: "Formatted at 2026-09-26 03:06:30 on dist-test-slave-hmbw"
server_key: "99eb5f589a0e28e4db1e23baba541578"
server_key_iv: "8d2f10d7796349ee1b4c2aafa83ea45d"
server_key_version: "encryptionkey@0"
I20260926 03:06:30.768923 5124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:30.774887 5124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:30.775729 5124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:30.775980 5124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:30.776932 5124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:06:30.778389 5124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:30.778530 5124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:30.778646 5124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:30.778714 5124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:30.809715 5124 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35729
I20260926 03:06:30.809791 5250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35729 every 8 connection(s)
I20260926 03:06:30.811054 5124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:06:30.812919 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 5124
I20260926 03:06:30.813113 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:06:30.813726 388 external_mini_cluster.cc:1428] Setting key b3c17572b02402cef1340990907e3f52
I20260926 03:06:30.816874 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42805
--builtin_ntp_servers=127.0.97.20:46091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 03:06:30.819799 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:30.813773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56469 (local address 127.0.97.62:42805)
0926 03:06:30.814266 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:30.814272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:30.815847 (+ 1575us) server_negotiation.cc:407] Connection header received
0926 03:06:30.817576 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:30.817581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:30.817646 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:30.817742 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:30.819101 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:30.819107 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:30.819112 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:06:30.819132 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:30.819171 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:06:30.819183 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:30.819298 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:30.819461 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 03:06:30.819527 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":234,"threads_started":1}
I20260926 03:06:30.820896 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:30.814535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42805 (local address 127.0.97.1:56469)
0926 03:06:30.815636 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:30.815684 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:30.817307 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:30.817999 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:30.818007 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:30.818108 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:30.818808 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:06:30.818810 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:30.818849 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:06:30.818939 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:30.819267 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:30.819277 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:30.819284 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 03:06:30.819539 (+ 255us) client_negotiation.cc:240] Negotiation successful
0926 03:06:30.819827 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":427,"threads_started":1}
I20260926 03:06:30.822620 5251 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42805
I20260926 03:06:30.822927 5251 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:30.823557 5251 heartbeater.cc:511] Master 127.0.97.62:42805 requested a full tablet report, sending...
I20260926 03:06:30.825767 5065 ts_manager.cc:194] Registered new tserver with Master: 1590e75dbe034536987f9f2e2c70e453 (127.0.97.1:35729)
W20260926 03:06:31.140251 5255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:31.140549 5255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:31.140661 5255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:31.148051 5255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:31.148164 5255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:31.148298 5255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:06:31.157524 5255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:31.158828 5255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:31.160655 5255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:31.174312 5261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:31.174283 5263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:31.174376 5260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:31.180430 5255 server_base.cc:1034] running on GCE node
I20260926 03:06:31.181092 5255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:31.182180 5255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:31.183403 5255 hybrid_clock.cc:648] HybridClock initialized: now 1790391991183358 us; error 36 us; skew 500 ppm
I20260926 03:06:31.183795 5255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:06:31.186193 5255 webserver.cc:460] Webserver started at http://127.0.97.2:42733/ using document root <none> and password file <none>
I20260926 03:06:31.186965 5255 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:31.187058 5255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:31.187367 5255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:31.190254 5255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-1/data/instance:
uuid: "fda63c0849114ef784099fca01934d2c"
format_stamp: "Formatted at 2026-09-26 03:06:31 on dist-test-slave-hmbw"
server_key: "e8bf1b539c371e0015298c0d8286dd8c"
server_key_iv: "a4b4fe9caa0a258d045e5907fa611f80"
server_key_version: "encryptionkey@0"
I20260926 03:06:31.191059 5255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fda63c0849114ef784099fca01934d2c"
format_stamp: "Formatted at 2026-09-26 03:06:31 on dist-test-slave-hmbw"
server_key: "e8bf1b539c371e0015298c0d8286dd8c"
server_key_iv: "a4b4fe9caa0a258d045e5907fa611f80"
server_key_version: "encryptionkey@0"
I20260926 03:06:31.196223 5255 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:06:31.200049 5269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:31.201555 5255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 03:06:31.201721 5255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "fda63c0849114ef784099fca01934d2c"
format_stamp: "Formatted at 2026-09-26 03:06:31 on dist-test-slave-hmbw"
server_key: "e8bf1b539c371e0015298c0d8286dd8c"
server_key_iv: "a4b4fe9caa0a258d045e5907fa611f80"
server_key_version: "encryptionkey@0"
I20260926 03:06:31.201956 5255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:31.208328 5255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:31.209357 5255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:31.209648 5255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:31.210703 5255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:06:31.212261 5255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:31.212360 5255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:31.212486 5255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:31.212553 5255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:31.243512 5255 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34517
I20260926 03:06:31.243556 5381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34517 every 8 connection(s)
I20260926 03:06:31.244861 5255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:06:31.246310 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 5255
I20260926 03:06:31.246457 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:06:31.246919 388 external_mini_cluster.cc:1428] Setting key c2953179b61d342a3f03a627a8acf7a6
I20260926 03:06:31.250381 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42805
--builtin_ntp_servers=127.0.97.20:46091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 03:06:31.254895 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:31.247566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46519 (local address 127.0.97.62:42805)
0926 03:06:31.247757 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:31.247762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:31.249598 (+ 1836us) server_negotiation.cc:407] Connection header received
0926 03:06:31.252095 (+ 2497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:31.252100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:31.252156 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:31.252253 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:31.253978 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:31.253985 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:31.253990 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:06:31.254014 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:31.254053 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:06:31.254069 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:31.254161 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:31.254485 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 03:06:31.254588 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:06:31.256871 5384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:31.248181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42805 (local address 127.0.97.2:46519)
0926 03:06:31.249437 (+ 1256us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:31.249488 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:31.251708 (+ 2220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:31.252556 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:31.252568 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:31.252712 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:31.253562 (+ 850us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:06:31.253565 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:31.253621 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:06:31.253752 (+ 131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:31.254249 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:31.254260 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:31.254267 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 03:06:31.255230 (+ 963us) client_negotiation.cc:240] Negotiation successful
0926 03:06:31.255549 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":420,"threads_started":1}
I20260926 03:06:31.258668 5382 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42805
I20260926 03:06:31.259069 5382 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:31.259696 5382 heartbeater.cc:511] Master 127.0.97.62:42805 requested a full tablet report, sending...
I20260926 03:06:31.261409 5065 ts_manager.cc:194] Registered new tserver with Master: fda63c0849114ef784099fca01934d2c (127.0.97.2:34517)
W20260926 03:06:31.585107 5385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:31.585424 5385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:31.585547 5385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:31.593050 5385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:31.593180 5385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:31.593325 5385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:06:31.602773 5385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:31.604045 5385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:31.605811 5385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:31.619051 5393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:31.619086 5391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:31.619024 5390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:31.619769 5385 server_base.cc:1034] running on GCE node
I20260926 03:06:31.620389 5385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:31.621474 5385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:31.622678 5385 hybrid_clock.cc:648] HybridClock initialized: now 1790391991622611 us; error 67 us; skew 500 ppm
I20260926 03:06:31.623055 5385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:06:31.625485 5385 webserver.cc:460] Webserver started at http://127.0.97.3:40709/ using document root <none> and password file <none>
I20260926 03:06:31.626219 5385 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:31.626323 5385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:31.626688 5385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:31.629823 5385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-2/data/instance:
uuid: "a1568d38b2e345d8bf6827200e65340a"
format_stamp: "Formatted at 2026-09-26 03:06:31 on dist-test-slave-hmbw"
server_key: "9f951af48dd169829faf7bca97ba15d7"
server_key_iv: "a5e38ab2bfa2f5dd30c197507f50c34d"
server_key_version: "encryptionkey@0"
I20260926 03:06:31.630718 5385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a1568d38b2e345d8bf6827200e65340a"
format_stamp: "Formatted at 2026-09-26 03:06:31 on dist-test-slave-hmbw"
server_key: "9f951af48dd169829faf7bca97ba15d7"
server_key_iv: "a5e38ab2bfa2f5dd30c197507f50c34d"
server_key_version: "encryptionkey@0"
I20260926 03:06:31.636137 5385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 03:06:31.640633 5399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:31.642848 5385 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 03:06:31.643072 5385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "a1568d38b2e345d8bf6827200e65340a"
format_stamp: "Formatted at 2026-09-26 03:06:31 on dist-test-slave-hmbw"
server_key: "9f951af48dd169829faf7bca97ba15d7"
server_key_iv: "a5e38ab2bfa2f5dd30c197507f50c34d"
server_key_version: "encryptionkey@0"
I20260926 03:06:31.643337 5385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:31.649513 5385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:31.650411 5385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:31.650732 5385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:31.651685 5385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:06:31.653177 5385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:31.653278 5385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:31.653390 5385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:31.653456 5385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:31.685570 5385 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40269
I20260926 03:06:31.685582 5511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40269 every 8 connection(s)
I20260926 03:06:31.687073 5385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:06:31.692447 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 5385
I20260926 03:06:31.692620 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:06:31.693158 388 external_mini_cluster.cc:1428] Setting key b5bf30dea7fb43a8b58551e0bd903ffd
I20260926 03:06:31.696725 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:31.690085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48357 (local address 127.0.97.62:42805)
0926 03:06:31.690344 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:31.690351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:31.691941 (+ 1590us) server_negotiation.cc:407] Connection header received
0926 03:06:31.694165 (+ 2224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:31.694170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:31.694236 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:31.694339 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:31.695906 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:31.695918 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:31.695923 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:06:31.695947 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:31.695985 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:06:31.696001 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:31.696123 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:31.696375 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 03:06:31.696428 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260926 03:06:31.697712 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:31.690713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42805 (local address 127.0.97.3:48357)
0926 03:06:31.691765 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:31.691804 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:31.693849 (+ 2045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:31.694683 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:31.694695 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:31.694840 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:31.695587 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:06:31.695589 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:31.695630 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:06:31.695724 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:31.696127 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:31.696139 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:31.696148 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 03:06:31.696446 (+ 298us) client_negotiation.cc:240] Negotiation successful
0926 03:06:31.696715 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":307,"threads_started":1}
I20260926 03:06:31.699134 5512 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42805
I20260926 03:06:31.699455 5512 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:31.700040 5512 heartbeater.cc:511] Master 127.0.97.62:42805 requested a full tablet report, sending...
I20260926 03:06:31.701506 5065 ts_manager.cc:194] Registered new tserver with Master: a1568d38b2e345d8bf6827200e65340a (127.0.97.3:40269)
I20260926 03:06:31.710227 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:06:31.717828 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:31.715250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51744 (local address 127.0.97.62:42805)
0926 03:06:31.715484 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:31.715491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:31.715751 (+ 260us) server_negotiation.cc:407] Connection header received
0926 03:06:31.716060 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:31.716064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:31.716121 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:31.716201 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:31.717147 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:31.717152 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:31.717155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:06:31.717172 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:31.717204 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:06:31.717216 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:31.717333 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:31.717413 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 03:06:31.717467 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 03:06:31.722026 5065 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:06:31.729080 5065 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:06:31.758854 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:31.742719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43170 (local address 127.0.97.2:34517)
0926 03:06:31.752895 (+ 10176us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:31.752907 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:31.755545 (+ 2638us) server_negotiation.cc:407] Connection header received
0926 03:06:31.755628 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:31.755636 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:31.755868 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:31.756015 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:31.757893 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:31.757904 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:31.757921 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:06:31.757940 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:31.758004 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:06:31.758032 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:31.758168 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:31.758324 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 03:06:31.758442 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9943,"thread_start_us":10673,"threads_started":1}
I20260926 03:06:31.758894 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:31.742036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34517 (local address 127.0.0.1:43170)
0926 03:06:31.755291 (+ 13255us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:31.755312 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:31.755417 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:31.756134 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:31.756144 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:31.756241 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:31.757601 (+ 1360us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:06:31.757607 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:31.757631 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:06:31.757713 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:31.758135 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:31.758146 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:31.758153 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 03:06:31.758332 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 03:06:31.758434 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13104,"thread_start_us":11883,"threads_started":1}
I20260926 03:06:31.759881 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:31.741870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40269 (local address 127.0.0.1:32780)
0926 03:06:31.754421 (+ 12551us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:31.754478 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:31.754765 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:31.757414 (+ 2649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:31.757419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:31.757443 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:31.758238 (+ 795us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:06:31.758242 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:31.758259 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:06:31.758302 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:31.759257 (+ 955us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:31.759264 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:31.759270 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 03:06:31.759394 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 03:06:31.759445 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12219,"thread_start_us":13359,"threads_started":1}
I20260926 03:06:31.760883 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:31.742013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32780 (local address 127.0.97.3:40269)
0926 03:06:31.755356 (+ 13343us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:31.755364 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:31.755401 (+ 37us) server_negotiation.cc:407] Connection header received
0926 03:06:31.755474 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:31.755483 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:31.755764 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:31.755928 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:31.758525 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:31.758536 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:31.758641 (+ 105us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:06:31.758672 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:31.758737 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:06:31.758823 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:31.760280 (+ 1457us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:31.760399 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 03:06:31.760502 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13145,"thread_start_us":12991,"threads_started":1}
I20260926 03:06:31.762036 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:31.755960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52828 (local address 127.0.97.1:35729)
0926 03:06:31.757424 (+ 1464us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:31.757433 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:31.758666 (+ 1233us) server_negotiation.cc:407] Connection header received
0926 03:06:31.758754 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:31.758764 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:31.759171 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:31.759353 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:31.761336 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:31.761343 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:31.761353 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:06:31.761367 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:31.761424 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:06:31.761446 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:31.761570 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:31.761637 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 03:06:31.761723 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1299,"spinlock_wait_cycles":1408,"thread_start_us":1280,"threads_started":1}
I20260926 03:06:31.762791 5529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:31.756790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35729 (local address 127.0.0.1:52828)
0926 03:06:31.758155 (+ 1365us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:31.758188 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:31.758344 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:31.759376 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:31.759380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:31.759401 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:31.760883 (+ 1482us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:06:31.760886 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:31.760901 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:06:31.761144 (+ 243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:31.761525 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:31.761530 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:31.761534 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 03:06:31.761651 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 03:06:31.761700 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1014,"thread_start_us":1040,"threads_started":1}
I20260926 03:06:31.763073 5317 tablet_service.cc:1467] Processing CreateTablet for tablet 24ff113616084852996616502a262d17 (DEFAULT_TABLE table=test-table [id=6b34d3003c874d4684553935494d601b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:31.764467 5317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24ff113616084852996616502a262d17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:31.764367 5447 tablet_service.cc:1467] Processing CreateTablet for tablet 24ff113616084852996616502a262d17 (DEFAULT_TABLE table=test-table [id=6b34d3003c874d4684553935494d601b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:31.765815 5447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24ff113616084852996616502a262d17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:31.766635 5186 tablet_service.cc:1467] Processing CreateTablet for tablet 24ff113616084852996616502a262d17 (DEFAULT_TABLE table=test-table [id=6b34d3003c874d4684553935494d601b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:31.768039 5186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24ff113616084852996616502a262d17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:31.778092 5530 tablet_bootstrap.cc:492] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c: Bootstrap starting.
I20260926 03:06:31.781319 5531 tablet_bootstrap.cc:492] T 24ff113616084852996616502a262d17 P a1568d38b2e345d8bf6827200e65340a: Bootstrap starting.
I20260926 03:06:31.782171 5530 tablet_bootstrap.cc:654] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:31.783493 5532 tablet_bootstrap.cc:492] T 24ff113616084852996616502a262d17 P 1590e75dbe034536987f9f2e2c70e453: Bootstrap starting.
I20260926 03:06:31.784876 5531 tablet_bootstrap.cc:654] T 24ff113616084852996616502a262d17 P a1568d38b2e345d8bf6827200e65340a: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:31.784970 5530 log.cc:824] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:31.786083 5531 log.cc:824] T 24ff113616084852996616502a262d17 P a1568d38b2e345d8bf6827200e65340a: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:31.786690 5532 tablet_bootstrap.cc:654] T 24ff113616084852996616502a262d17 P 1590e75dbe034536987f9f2e2c70e453: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:31.787808 5532 log.cc:824] T 24ff113616084852996616502a262d17 P 1590e75dbe034536987f9f2e2c70e453: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:31.788251 5530 tablet_bootstrap.cc:492] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c: No bootstrap required, opened a new log
I20260926 03:06:31.788543 5530 ts_tablet_manager.cc:1397] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260926 03:06:31.789048 5531 tablet_bootstrap.cc:492] T 24ff113616084852996616502a262d17 P a1568d38b2e345d8bf6827200e65340a: No bootstrap required, opened a new log
I20260926 03:06:31.789312 5531 ts_tablet_manager.cc:1397] T 24ff113616084852996616502a262d17 P a1568d38b2e345d8bf6827200e65340a: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260926 03:06:31.790309 5532 tablet_bootstrap.cc:492] T 24ff113616084852996616502a262d17 P 1590e75dbe034536987f9f2e2c70e453: No bootstrap required, opened a new log
I20260926 03:06:31.790593 5532 ts_tablet_manager.cc:1397] T 24ff113616084852996616502a262d17 P 1590e75dbe034536987f9f2e2c70e453: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.005s
I20260926 03:06:31.799093 5531 raft_consensus.cc:348] T 24ff113616084852996616502a262d17 P a1568d38b2e345d8bf6827200e65340a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1568d38b2e345d8bf6827200e65340a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40269 } } peers { permanent_uuid: "fda63c0849114ef784099fca01934d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34517 } } peers { permanent_uuid: "1590e75dbe034536987f9f2e2c70e453" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35729 } }
I20260926 03:06:31.799695 5531 raft_consensus.cc:374] T 24ff113616084852996616502a262d17 P a1568d38b2e345d8bf6827200e65340a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:31.799808 5531 raft_consensus.cc:729] T 24ff113616084852996616502a262d17 P a1568d38b2e345d8bf6827200e65340a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1568d38b2e345d8bf6827200e65340a, State: Initialized, Role: FOLLOWER
I20260926 03:06:31.800657 5531 consensus_queue.cc:260] T 24ff113616084852996616502a262d17 P a1568d38b2e345d8bf6827200e65340a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1568d38b2e345d8bf6827200e65340a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40269 } } peers { permanent_uuid: "fda63c0849114ef784099fca01934d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34517 } } peers { permanent_uuid: "1590e75dbe034536987f9f2e2c70e453" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35729 } }
I20260926 03:06:31.800721 5530 raft_consensus.cc:348] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1568d38b2e345d8bf6827200e65340a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40269 } } peers { permanent_uuid: "fda63c0849114ef784099fca01934d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34517 } } peers { permanent_uuid: "1590e75dbe034536987f9f2e2c70e453" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35729 } }
I20260926 03:06:31.800645 5532 raft_consensus.cc:348] T 24ff113616084852996616502a262d17 P 1590e75dbe034536987f9f2e2c70e453 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1568d38b2e345d8bf6827200e65340a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40269 } } peers { permanent_uuid: "fda63c0849114ef784099fca01934d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34517 } } peers { permanent_uuid: "1590e75dbe034536987f9f2e2c70e453" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35729 } }
I20260926 03:06:31.801182 5530 raft_consensus.cc:374] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:31.801190 5532 raft_consensus.cc:374] T 24ff113616084852996616502a262d17 P 1590e75dbe034536987f9f2e2c70e453 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:31.801280 5532 raft_consensus.cc:729] T 24ff113616084852996616502a262d17 P 1590e75dbe034536987f9f2e2c70e453 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1590e75dbe034536987f9f2e2c70e453, State: Initialized, Role: FOLLOWER
I20260926 03:06:31.801280 5530 raft_consensus.cc:729] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fda63c0849114ef784099fca01934d2c, State: Initialized, Role: FOLLOWER
I20260926 03:06:31.802044 5512 heartbeater.cc:503] Master 127.0.97.62:42805 was elected leader, sending a full tablet report...
I20260926 03:06:31.801945 5530 consensus_queue.cc:260] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1568d38b2e345d8bf6827200e65340a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40269 } } peers { permanent_uuid: "fda63c0849114ef784099fca01934d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34517 } } peers { permanent_uuid: "1590e75dbe034536987f9f2e2c70e453" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35729 } }
I20260926 03:06:31.802057 5532 consensus_queue.cc:260] T 24ff113616084852996616502a262d17 P 1590e75dbe034536987f9f2e2c70e453 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1568d38b2e345d8bf6827200e65340a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40269 } } peers { permanent_uuid: "fda63c0849114ef784099fca01934d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34517 } } peers { permanent_uuid: "1590e75dbe034536987f9f2e2c70e453" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35729 } }
I20260926 03:06:31.803115 5382 heartbeater.cc:503] Master 127.0.97.62:42805 was elected leader, sending a full tablet report...
I20260926 03:06:31.803908 5530 ts_tablet_manager.cc:1428] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260926 03:06:31.804790 5251 heartbeater.cc:503] Master 127.0.97.62:42805 was elected leader, sending a full tablet report...
I20260926 03:06:31.805493 5531 ts_tablet_manager.cc:1428] T 24ff113616084852996616502a262d17 P a1568d38b2e345d8bf6827200e65340a: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
I20260926 03:06:31.806463 5532 ts_tablet_manager.cc:1428] T 24ff113616084852996616502a262d17 P 1590e75dbe034536987f9f2e2c70e453: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
W20260926 03:06:31.813223 5252 tablet.cc:2369] T 24ff113616084852996616502a262d17 P 1590e75dbe034536987f9f2e2c70e453: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:06:31.939000 5513 tablet.cc:2369] T 24ff113616084852996616502a262d17 P a1568d38b2e345d8bf6827200e65340a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:06:31.958163 5537 raft_consensus.cc:482] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:06:31.958463 5537 raft_consensus.cc:504] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1568d38b2e345d8bf6827200e65340a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40269 } } peers { permanent_uuid: "fda63c0849114ef784099fca01934d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34517 } } peers { permanent_uuid: "1590e75dbe034536987f9f2e2c70e453" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35729 } }
I20260926 03:06:31.960309 5537 leader_election.cc:288] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1568d38b2e345d8bf6827200e65340a (127.0.97.3:40269), 1590e75dbe034536987f9f2e2c70e453 (127.0.97.1:35729)
I20260926 03:06:31.963768 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:31.960852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40085 (local address 127.0.97.3:40269)
0926 03:06:31.961052 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:31.961060 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:31.961590 (+ 530us) server_negotiation.cc:407] Connection header received
0926 03:06:31.961755 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:31.961761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:31.961814 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:31.961910 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:31.963176 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:31.963187 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:31.963190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:06:31.963210 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:31.963250 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:06:31.963266 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:31.963399 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:31.963507 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 03:06:31.963559 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:06:31.963996 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:31.960791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40269 (local address 127.0.97.2:40085)
0926 03:06:31.961479 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:31.961494 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:31.961640 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:31.962090 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:31.962094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:31.962129 (+ 35us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:31.962918 (+ 789us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:06:31.962921 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:31.962942 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:06:31.962996 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:31.963383 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:31.963388 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:31.963393 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:06:31.963546 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 03:06:31.963599 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":383,"threads_started":1}
I20260926 03:06:31.964566 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:31.961829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51221 (local address 127.0.97.1:35729)
0926 03:06:31.962009 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:31.962014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:31.962356 (+ 342us) server_negotiation.cc:407] Connection header received
0926 03:06:31.962467 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:31.962472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:31.962558 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:31.962663 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:31.963814 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:31.963820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:31.963824 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:06:31.963844 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:31.963880 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:06:31.963896 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:31.964007 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:31.964236 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 03:06:31.964312 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 03:06:31.964583 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:31.961623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35729 (local address 127.0.97.2:51221)
0926 03:06:31.962228 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:31.962252 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:31.962366 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:31.962844 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:31.962850 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:31.962882 (+ 32us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:31.963567 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:06:31.963570 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:31.963585 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:06:31.963622 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:31.964050 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:31.964055 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:31.964060 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:06:31.964265 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 03:06:31.964304 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":258,"threads_started":1}
I20260926 03:06:31.965070 5467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24ff113616084852996616502a262d17" candidate_uuid: "fda63c0849114ef784099fca01934d2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1568d38b2e345d8bf6827200e65340a" is_pre_election: true
I20260926 03:06:31.965626 5467 raft_consensus.cc:2454] T 24ff113616084852996616502a262d17 P a1568d38b2e345d8bf6827200e65340a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fda63c0849114ef784099fca01934d2c in term 0.
I20260926 03:06:31.965711 5206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24ff113616084852996616502a262d17" candidate_uuid: "fda63c0849114ef784099fca01934d2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1590e75dbe034536987f9f2e2c70e453" is_pre_election: true
I20260926 03:06:31.966217 5206 raft_consensus.cc:2454] T 24ff113616084852996616502a262d17 P 1590e75dbe034536987f9f2e2c70e453 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fda63c0849114ef784099fca01934d2c in term 0.
I20260926 03:06:31.966377 5270 leader_election.cc:302] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1568d38b2e345d8bf6827200e65340a, fda63c0849114ef784099fca01934d2c; no voters:
I20260926 03:06:31.966874 5537 raft_consensus.cc:2781] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:06:31.966991 5537 raft_consensus.cc:482] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:06:31.967085 5537 raft_consensus.cc:3037] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:31.969321 5537 raft_consensus.cc:504] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1568d38b2e345d8bf6827200e65340a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40269 } } peers { permanent_uuid: "fda63c0849114ef784099fca01934d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34517 } } peers { permanent_uuid: "1590e75dbe034536987f9f2e2c70e453" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35729 } }
I20260926 03:06:31.970067 5537 leader_election.cc:288] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c [CANDIDATE]: Term 1 election: Requested vote from peers a1568d38b2e345d8bf6827200e65340a (127.0.97.3:40269), 1590e75dbe034536987f9f2e2c70e453 (127.0.97.1:35729)
I20260926 03:06:31.970630 5467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24ff113616084852996616502a262d17" candidate_uuid: "fda63c0849114ef784099fca01934d2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1568d38b2e345d8bf6827200e65340a"
I20260926 03:06:31.970659 5206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24ff113616084852996616502a262d17" candidate_uuid: "fda63c0849114ef784099fca01934d2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1590e75dbe034536987f9f2e2c70e453"
I20260926 03:06:31.970831 5206 raft_consensus.cc:3037] T 24ff113616084852996616502a262d17 P 1590e75dbe034536987f9f2e2c70e453 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:31.970832 5467 raft_consensus.cc:3037] T 24ff113616084852996616502a262d17 P a1568d38b2e345d8bf6827200e65340a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:31.973137 5467 raft_consensus.cc:2454] T 24ff113616084852996616502a262d17 P a1568d38b2e345d8bf6827200e65340a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fda63c0849114ef784099fca01934d2c in term 1.
I20260926 03:06:31.973167 5206 raft_consensus.cc:2454] T 24ff113616084852996616502a262d17 P 1590e75dbe034536987f9f2e2c70e453 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fda63c0849114ef784099fca01934d2c in term 1.
I20260926 03:06:31.973608 5270 leader_election.cc:302] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1590e75dbe034536987f9f2e2c70e453, fda63c0849114ef784099fca01934d2c; no voters:
I20260926 03:06:31.974093 5537 raft_consensus.cc:2781] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:31.974567 5537 raft_consensus.cc:686] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c [term 1 LEADER]: Becoming Leader. State: Replica: fda63c0849114ef784099fca01934d2c, State: Running, Role: LEADER
I20260926 03:06:31.975206 5537 consensus_queue.cc:237] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1568d38b2e345d8bf6827200e65340a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40269 } } peers { permanent_uuid: "fda63c0849114ef784099fca01934d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34517 } } peers { permanent_uuid: "1590e75dbe034536987f9f2e2c70e453" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35729 } }
I20260926 03:06:31.979702 5063 catalog_manager.cc:5495] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c reported cstate change: term changed from 0 to 1, leader changed from <none> to fda63c0849114ef784099fca01934d2c (127.0.97.2). New cstate: current_term: 1 leader_uuid: "fda63c0849114ef784099fca01934d2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1568d38b2e345d8bf6827200e65340a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fda63c0849114ef784099fca01934d2c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1590e75dbe034536987f9f2e2c70e453" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35729 } health_report { overall_health: UNKNOWN } } }
W20260926 03:06:31.997051 5383 tablet.cc:2369] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:06:32.049966 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:32.047804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43178 (local address 127.0.97.2:34517)
0926 03:06:32.047992 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:32.047997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:32.048017 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:06:32.048214 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:32.048219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:32.048312 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:32.048403 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:32.049360 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:32.049366 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:32.049369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:06:32.049385 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:32.049416 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:06:32.049428 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:32.049533 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:32.049636 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 03:06:32.049697 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 03:06:32.058718 5206 raft_consensus.cc:1261] T 24ff113616084852996616502a262d17 P 1590e75dbe034536987f9f2e2c70e453 [term 1 FOLLOWER]: Refusing update from remote peer fda63c0849114ef784099fca01934d2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:32.058718 5467 raft_consensus.cc:1261] T 24ff113616084852996616502a262d17 P a1568d38b2e345d8bf6827200e65340a [term 1 FOLLOWER]: Refusing update from remote peer fda63c0849114ef784099fca01934d2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:32.059711 5541 consensus_queue.cc:1035] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1590e75dbe034536987f9f2e2c70e453" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:32.060160 5537 consensus_queue.cc:1035] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1568d38b2e345d8bf6827200e65340a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:32.073457 5544 mvcc.cc:205] Tried to move back new op lower bound from 7333445599461715968 to 7333445599134420992. Current Snapshot: MvccSnapshot[applied={T|T < 7333445599461715968}]
I20260926 03:06:32.079134 5545 mvcc.cc:205] Tried to move back new op lower bound from 7333445599461715968 to 7333445599134420992. Current Snapshot: MvccSnapshot[applied={T|T < 7333445599461715968}]
I20260926 03:06:32.093372 5063 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:06:32.093658 5063 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:06:32.097615 5063 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 18d2ee59b60e483fad71ae73925fc976: Sending DeleteTablet for 3 replicas of tablet 24ff113616084852996616502a262d17
I20260926 03:06:32.098899 5317 tablet_service.cc:1514] Processing DeleteTablet for tablet 24ff113616084852996616502a262d17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:32 UTC) from {username='slave'} at 127.0.0.1:43170
I20260926 03:06:32.099241 5186 tablet_service.cc:1514] Processing DeleteTablet for tablet 24ff113616084852996616502a262d17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:32 UTC) from {username='slave'} at 127.0.0.1:52828
I20260926 03:06:32.099361 5447 tablet_service.cc:1514] Processing DeleteTablet for tablet 24ff113616084852996616502a262d17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:32 UTC) from {username='slave'} at 127.0.0.1:32780
I20260926 03:06:32.100459 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 5124
I20260926 03:06:32.101915 5556 tablet_replica.cc:330] T 24ff113616084852996616502a262d17 P a1568d38b2e345d8bf6827200e65340a: stopping tablet replica
I20260926 03:06:32.102378 5556 raft_consensus.cc:2229] T 24ff113616084852996616502a262d17 P a1568d38b2e345d8bf6827200e65340a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:06:32.102937 5556 raft_consensus.cc:2258] T 24ff113616084852996616502a262d17 P a1568d38b2e345d8bf6827200e65340a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:32.103724 5555 tablet_replica.cc:330] T 24ff113616084852996616502a262d17 P 1590e75dbe034536987f9f2e2c70e453: stopping tablet replica
I20260926 03:06:32.104161 5555 raft_consensus.cc:2229] T 24ff113616084852996616502a262d17 P 1590e75dbe034536987f9f2e2c70e453 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:06:32.104669 5555 raft_consensus.cc:2258] T 24ff113616084852996616502a262d17 P 1590e75dbe034536987f9f2e2c70e453 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:32.105559 5556 ts_tablet_manager.cc:1905] T 24ff113616084852996616502a262d17 P a1568d38b2e345d8bf6827200e65340a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:32.107234 5555 ts_tablet_manager.cc:1905] T 24ff113616084852996616502a262d17 P 1590e75dbe034536987f9f2e2c70e453: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:32.111073 5554 tablet_replica.cc:330] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c: stopping tablet replica
I20260926 03:06:32.111474 5554 raft_consensus.cc:2229] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:06:32.112121 5554 raft_consensus.cc:2258] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:32.112675 5556 ts_tablet_manager.cc:1918] T 24ff113616084852996616502a262d17 P a1568d38b2e345d8bf6827200e65340a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:06:32.112848 5556 log.cc:1196] T 24ff113616084852996616502a262d17 P a1568d38b2e345d8bf6827200e65340a: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-2/wal/wals/24ff113616084852996616502a262d17
I20260926 03:06:32.113355 5556 ts_tablet_manager.cc:1939] T 24ff113616084852996616502a262d17 P a1568d38b2e345d8bf6827200e65340a: Deleting consensus metadata
I20260926 03:06:32.115195 5554 ts_tablet_manager.cc:1905] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:32.115295 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:32.102785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52844 (local address 127.0.97.1:35729)
0926 03:06:32.102967 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:32.102974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:32.112674 (+ 9700us) server_negotiation.cc:407] Connection header received
0926 03:06:32.113272 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:32.113278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:32.113397 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:32.113510 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:32.114530 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:32.114535 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:32.114540 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:06:32.114560 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:32.114596 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:06:32.114612 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:32.114723 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:32.114958 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 03:06:32.115019 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:06:32.115312 5049 catalog_manager.cc:4841] TS a1568d38b2e345d8bf6827200e65340a (127.0.97.3:40269): tablet 24ff113616084852996616502a262d17 (table test-table [id=6b34d3003c874d4684553935494d601b]) successfully deleted
I20260926 03:06:32.116243 5246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:32.124622 5554 ts_tablet_manager.cc:1918] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:06:32.124848 5554 log.cc:1196] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391950058021-388-0/minicluster-data/ts-1/wal/wals/24ff113616084852996616502a262d17
I20260926 03:06:32.125427 5554 ts_tablet_manager.cc:1939] T 24ff113616084852996616502a262d17 P fda63c0849114ef784099fca01934d2c: Deleting consensus metadata
I20260926 03:06:32.127429 5052 catalog_manager.cc:4841] TS fda63c0849114ef784099fca01934d2c (127.0.97.2:34517): tablet 24ff113616084852996616502a262d17 (table test-table [id=6b34d3003c874d4684553935494d601b]) successfully deleted
I20260926 03:06:32.207851 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 5124
W20260926 03:06:32.222095 5049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:35729 (error 108)
I20260926 03:06:32.222348 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 5255
I20260926 03:06:32.224306 5529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:32.223788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35729 (local address 127.0.0.1:52850)
0926 03:06:32.224039 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:32.224142 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260926 03:06:32.224561 5049 catalog_manager.cc:4584] TS 1590e75dbe034536987f9f2e2c70e453 (127.0.97.1:35729): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 24ff113616084852996616502a262d17: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35729: connect: Connection refused (error 111)
I20260926 03:06:32.225507 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:32.223143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43192 (local address 127.0.97.2:34517)
0926 03:06:32.223303 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:32.223309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:32.223428 (+ 119us) server_negotiation.cc:407] Connection header received
0926 03:06:32.223621 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:32.223626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:32.223693 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:32.223812 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:32.224888 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:32.224893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:32.224900 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:06:32.224916 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:32.224946 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:06:32.224958 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:32.225074 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:32.225175 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 03:06:32.225221 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 03:06:32.226320 5377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:32.278343 5529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:32.277825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35729 (local address 127.0.0.1:52864)
0926 03:06:32.278055 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:32.278175 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 03:06:32.279403 5529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:32.279040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35729 (local address 127.0.0.1:52870)
0926 03:06:32.279180 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:32.279264 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 03:06:32.339591 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 5255
I20260926 03:06:32.339802 5529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:32.339233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35729 (local address 127.0.0.1:52878)
0926 03:06:32.339412 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:32.339523 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 03:06:32.340929 5529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:32.340543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35729 (local address 127.0.0.1:52884)
0926 03:06:32.340701 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:32.340794 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 03:06:32.354728 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 5385
I20260926 03:06:32.357474 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:32.355425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32782 (local address 127.0.97.3:40269)
0926 03:06:32.355604 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:32.355609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:32.355626 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:06:32.355792 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:32.355796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:32.355882 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:32.355968 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:32.356781 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:32.356784 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:32.356788 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:06:32.356802 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:32.356833 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:06:32.356851 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:32.356959 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:32.357131 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 03:06:32.357187 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:06:32.358057 5507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:32.421563 5529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:32.421115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35729 (local address 127.0.0.1:52900)
0926 03:06:32.421320 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:32.421412 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 03:06:32.422737 5529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:32.422264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35729 (local address 127.0.0.1:52914)
0926 03:06:32.422462 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:32.422581 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260926 03:06:32.444813 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 5385
I20260926 03:06:32.458207 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 5034
I20260926 03:06:32.459473 5095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:32.564343 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 5034
2026-09-26T03:06:32Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2746 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 03:06:32.601413 388 mini_oidc.cc:152] Starting JWKS server
I20260926 03:06:32.604038 388 webserver.cc:460] Webserver started at https://127.0.0.1:34443/ using document root <none> and password file <none>
I20260926 03:06:32.604363 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 03:06:32.605077 388 webserver.cc:460] Webserver started at http://127.0.0.1:35301/ using document root <none> and password file <none>
2026-09-26T03:06:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:06:32Z Disabled control of system clock
I20260926 03:06:32.630028 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42893
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42893
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34443/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391950058021-388-0/testchainca.cert with env {}
W20260926 03:06:32.955185 5568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:32.955493 5568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:32.955593 5568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:32.962992 5568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:06:32.963112 5568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:32.963148 5568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:32.963202 5568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:06:32.963245 5568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 03:06:32.963310 5568 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 03:06:32.963348 5568 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34443/jwks.json
I20260926 03:06:32.973438 5568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42893
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34443/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391950058021-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:32.974759 5568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:32.976032 5568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:32.990931 5573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:32.991019 5576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:32.990984 5574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:32.992010 5568 server_base.cc:1034] running on GCE node
I20260926 03:06:32.992862 5568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:32.994462 5568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:32.995764 5568 hybrid_clock.cc:648] HybridClock initialized: now 1790391992995698 us; error 63 us; skew 500 ppm
I20260926 03:06:32.996143 5568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:06:32.998765 5568 webserver.cc:460] Webserver started at http://127.0.97.62:37755/ using document root <none> and password file <none>
I20260926 03:06:32.999493 5568 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:32.999621 5568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:32.999925 5568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:33.002997 5568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "cb8eaffca1ba450a8ac40e5075830f85"
format_stamp: "Formatted at 2026-09-26 03:06:32 on dist-test-slave-hmbw"
server_key: "616aec5b57ad9215f6dba6dacee1232d"
server_key_iv: "4b8f451d96221641a097a71b6eb5df42"
server_key_version: "encryptionkey@0"
I20260926 03:06:33.003826 5568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "cb8eaffca1ba450a8ac40e5075830f85"
format_stamp: "Formatted at 2026-09-26 03:06:32 on dist-test-slave-hmbw"
server_key: "616aec5b57ad9215f6dba6dacee1232d"
server_key_iv: "4b8f451d96221641a097a71b6eb5df42"
server_key_version: "encryptionkey@0"
I20260926 03:06:33.009506 5568 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260926 03:06:33.014343 5582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:33.016084 5568 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:06:33.016286 5568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "cb8eaffca1ba450a8ac40e5075830f85"
format_stamp: "Formatted at 2026-09-26 03:06:32 on dist-test-slave-hmbw"
server_key: "616aec5b57ad9215f6dba6dacee1232d"
server_key_iv: "4b8f451d96221641a097a71b6eb5df42"
server_key_version: "encryptionkey@0"
I20260926 03:06:33.016510 5568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:33.037523 5568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:33.038478 5568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:33.038796 5568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:33.057440 5568 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42893
I20260926 03:06:33.057441 5633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42893 every 8 connection(s)
I20260926 03:06:33.059346 5568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:06:33.063956 5634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:33.069346 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 5568
I20260926 03:06:33.069571 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:06:33.070365 388 external_mini_cluster.cc:1428] Setting key 4b40c6717d87b83fdcf18cf0e4cb0907
I20260926 03:06:33.076640 5634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb8eaffca1ba450a8ac40e5075830f85: Bootstrap starting.
I20260926 03:06:33.080399 5634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb8eaffca1ba450a8ac40e5075830f85: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:33.081598 5634 log.cc:824] T 00000000000000000000000000000000 P cb8eaffca1ba450a8ac40e5075830f85: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:33.084841 5634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb8eaffca1ba450a8ac40e5075830f85: No bootstrap required, opened a new log
I20260926 03:06:33.085680 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:33.072419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41270 (local address 127.0.97.62:42893)
0926 03:06:33.073547 (+ 1128us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:33.073559 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:33.073608 (+ 49us) server_negotiation.cc:407] Connection header received
0926 03:06:33.075481 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:33.075509 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:33.076061 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:33.076474 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:33.077132 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:33.078847 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:33.080857 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:33.081625 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:33.081845 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:33.083165 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:33.083174 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:33.083194 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:06:33.083763 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:33.083846 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:06:33.083868 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:33.083964 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:33.084247 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 03:06:33.084604 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":418,"threads_started":1}
I20260926 03:06:33.090104 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:06:33.094578 5634 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb8eaffca1ba450a8ac40e5075830f85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8eaffca1ba450a8ac40e5075830f85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42893 } }
I20260926 03:06:33.094966 5634 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb8eaffca1ba450a8ac40e5075830f85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:33.095039 5634 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb8eaffca1ba450a8ac40e5075830f85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb8eaffca1ba450a8ac40e5075830f85, State: Initialized, Role: FOLLOWER
I20260926 03:06:33.095549 5634 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb8eaffca1ba450a8ac40e5075830f85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8eaffca1ba450a8ac40e5075830f85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42893 } }
I20260926 03:06:33.095755 5634 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb8eaffca1ba450a8ac40e5075830f85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:06:33.095851 5634 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb8eaffca1ba450a8ac40e5075830f85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:06:33.095973 5634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb8eaffca1ba450a8ac40e5075830f85 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:33.098129 5634 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb8eaffca1ba450a8ac40e5075830f85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8eaffca1ba450a8ac40e5075830f85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42893 } }
I20260926 03:06:33.098706 5634 leader_election.cc:302] T 00000000000000000000000000000000 P cb8eaffca1ba450a8ac40e5075830f85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb8eaffca1ba450a8ac40e5075830f85; no voters:
I20260926 03:06:33.099319 5634 leader_election.cc:288] T 00000000000000000000000000000000 P cb8eaffca1ba450a8ac40e5075830f85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:06:33.099512 5639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb8eaffca1ba450a8ac40e5075830f85 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:33.100492 5639 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb8eaffca1ba450a8ac40e5075830f85 [term 1 LEADER]: Becoming Leader. State: Replica: cb8eaffca1ba450a8ac40e5075830f85, State: Running, Role: LEADER
I20260926 03:06:33.101022 5639 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb8eaffca1ba450a8ac40e5075830f85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8eaffca1ba450a8ac40e5075830f85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42893 } }
I20260926 03:06:33.101441 5634 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb8eaffca1ba450a8ac40e5075830f85 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:06:33.103889 5641 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb8eaffca1ba450a8ac40e5075830f85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb8eaffca1ba450a8ac40e5075830f85. Latest consensus state: current_term: 1 leader_uuid: "cb8eaffca1ba450a8ac40e5075830f85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8eaffca1ba450a8ac40e5075830f85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42893 } } }
I20260926 03:06:33.104317 5641 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb8eaffca1ba450a8ac40e5075830f85 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:33.104651 5640 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb8eaffca1ba450a8ac40e5075830f85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb8eaffca1ba450a8ac40e5075830f85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8eaffca1ba450a8ac40e5075830f85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42893 } } }
I20260926 03:06:33.105034 5640 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb8eaffca1ba450a8ac40e5075830f85 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:33.106691 5650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:06:33.111820 5650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:06:33.121378 5650 catalog_manager.cc:1347] Generated new cluster ID: 17bb1cbffcf8421396ef6a2fbda807b6
I20260926 03:06:33.121488 5650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:06:33.132112 5650 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:06:33.134075 5650 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:06:33.149468 5650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb8eaffca1ba450a8ac40e5075830f85: Generated new TSK 0
I20260926 03:06:33.150705 5650 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 03:06:33.201268 5664 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 03:06:33.201471 5664 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 03:06:33.202538 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:33.180239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41280 (local address 127.0.97.62:42893)
0926 03:06:33.180451 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:33.180459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:33.181022 (+ 563us) server_negotiation.cc:407] Connection header received
0926 03:06:33.181376 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:33.181381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:33.181456 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:33.181581 (+ 125us) server_negotiation.cc:226] Negotiated authn=JWT
0926 03:06:33.182203 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:33.183057 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:33.185167 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:33.185649 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:33.185768 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:33.186457 (+ 689us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 03:06:33.201914 (+ 15457us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 03:06:33.202245 (+ 331us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41280: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 03:06:33.202679 5637 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41280: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 03:06:33.204666 5663 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:06:33.180206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42893 (local address 127.0.0.1:41280)
0926 03:06:33.180895 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:33.180916 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:33.181113 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:33.181642 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:33.181648 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:33.181661 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0926 03:06:33.182029 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:33.182041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:33.183206 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:33.183212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:33.184979 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:33.184996 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:33.185792 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:33.185840 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:33.186044 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:33.186137 (+ 93us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 03:06:33.202251 (+ 16114us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 03:06:33.202795 (+ 544us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:42893: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":337,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:42893: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 03:06:33.206717 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 5568
I20260926 03:06:33.208086 5629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:33.302922 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 5568
2026-09-26T03:06:33Z chronyd exiting
I20260926 03:06:33.349994 388 test_util.cc:175] -----------------------------------------------
I20260926 03:06:33.350143 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391950058021-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (753 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 03:06:33.353904 388 mini_oidc.cc:152] Starting JWKS server
I20260926 03:06:33.355751 388 webserver.cc:460] Webserver started at https://127.0.0.1:46855/ using document root <none> and password file <none>
I20260926 03:06:33.355965 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 03:06:33.356600 388 webserver.cc:460] Webserver started at http://127.0.0.1:41599/ using document root <none> and password file <none>
2026-09-26T03:06:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:06:33Z Disabled control of system clock
I20260926 03:06:33.381043 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37425
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37425
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46855/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391950058021-388-0/testchainca.cert with env {}
W20260926 03:06:33.700811 5671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:33.701120 5671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:33.701205 5671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:33.708603 5671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:06:33.708722 5671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:33.708758 5671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:33.708806 5671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:06:33.708849 5671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 03:06:33.708909 5671 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 03:06:33.708946 5671 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46855/jwks.json
I20260926 03:06:33.719316 5671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37425
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46855/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391950058021-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:33.720573 5671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:33.721796 5671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:33.733389 5679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:33.733462 5677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:33.733479 5676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:33.734014 5671 server_base.cc:1034] running on GCE node
I20260926 03:06:33.734858 5671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:33.736315 5671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:33.737542 5671 hybrid_clock.cc:648] HybridClock initialized: now 1790391993737500 us; error 36 us; skew 500 ppm
I20260926 03:06:33.737916 5671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:06:33.740329 5671 webserver.cc:460] Webserver started at http://127.0.97.62:35007/ using document root <none> and password file <none>
I20260926 03:06:33.741030 5671 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:33.741137 5671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:33.741446 5671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:33.744311 5671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "08267c74e3ab46669a3bf0ded8825967"
format_stamp: "Formatted at 2026-09-26 03:06:33 on dist-test-slave-hmbw"
server_key: "cc33bbc55ca47439a1b5716347183120"
server_key_iv: "8707872e0709cf841ef0fd655b4c2802"
server_key_version: "encryptionkey@0"
I20260926 03:06:33.745102 5671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "08267c74e3ab46669a3bf0ded8825967"
format_stamp: "Formatted at 2026-09-26 03:06:33 on dist-test-slave-hmbw"
server_key: "cc33bbc55ca47439a1b5716347183120"
server_key_iv: "8707872e0709cf841ef0fd655b4c2802"
server_key_version: "encryptionkey@0"
I20260926 03:06:33.750136 5671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:06:33.753664 5685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:33.755101 5671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 03:06:33.755272 5671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "08267c74e3ab46669a3bf0ded8825967"
format_stamp: "Formatted at 2026-09-26 03:06:33 on dist-test-slave-hmbw"
server_key: "cc33bbc55ca47439a1b5716347183120"
server_key_iv: "8707872e0709cf841ef0fd655b4c2802"
server_key_version: "encryptionkey@0"
I20260926 03:06:33.755470 5671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:33.769845 5671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:33.770761 5671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:33.771020 5671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:33.788702 5671 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37425
I20260926 03:06:33.788721 5736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37425 every 8 connection(s)
I20260926 03:06:33.790081 5671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:06:33.793996 5737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:33.799556 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 5671
I20260926 03:06:33.799748 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:06:33.800489 388 external_mini_cluster.cc:1428] Setting key e61991ef768e5e138b9f5b496d321b0a
I20260926 03:06:33.806257 5737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08267c74e3ab46669a3bf0ded8825967: Bootstrap starting.
I20260926 03:06:33.810141 5737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08267c74e3ab46669a3bf0ded8825967: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:33.811614 5737 log.cc:824] T 00000000000000000000000000000000 P 08267c74e3ab46669a3bf0ded8825967: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:33.815565 5737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08267c74e3ab46669a3bf0ded8825967: No bootstrap required, opened a new log
I20260926 03:06:33.817232 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:33.802650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56668 (local address 127.0.97.62:37425)
0926 03:06:33.803934 (+ 1284us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:33.803950 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:33.803994 (+ 44us) server_negotiation.cc:407] Connection header received
0926 03:06:33.805693 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:33.805723 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:33.806231 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:33.806675 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:33.807210 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:33.809027 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:33.810883 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:33.812019 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:33.812215 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:33.813806 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:33.813818 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:33.813837 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:06:33.814604 (+ 767us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:33.814744 (+ 140us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:06:33.814775 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:33.814909 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:33.815329 (+ 420us) server_negotiation.cc:299] Negotiation successful
0926 03:06:33.815755 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":454,"threads_started":1}
I20260926 03:06:33.821343 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:06:33.824447 5737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08267c74e3ab46669a3bf0ded8825967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08267c74e3ab46669a3bf0ded8825967" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } }
I20260926 03:06:33.824831 5737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08267c74e3ab46669a3bf0ded8825967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:33.824926 5737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08267c74e3ab46669a3bf0ded8825967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08267c74e3ab46669a3bf0ded8825967, State: Initialized, Role: FOLLOWER
I20260926 03:06:33.825497 5737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08267c74e3ab46669a3bf0ded8825967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08267c74e3ab46669a3bf0ded8825967" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } }
I20260926 03:06:33.825713 5737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08267c74e3ab46669a3bf0ded8825967 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:06:33.825835 5737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08267c74e3ab46669a3bf0ded8825967 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:06:33.825987 5737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08267c74e3ab46669a3bf0ded8825967 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:33.828137 5737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08267c74e3ab46669a3bf0ded8825967 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08267c74e3ab46669a3bf0ded8825967" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } }
I20260926 03:06:33.828624 5737 leader_election.cc:302] T 00000000000000000000000000000000 P 08267c74e3ab46669a3bf0ded8825967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08267c74e3ab46669a3bf0ded8825967; no voters:
I20260926 03:06:33.829162 5737 leader_election.cc:288] T 00000000000000000000000000000000 P 08267c74e3ab46669a3bf0ded8825967 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:06:33.829310 5742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08267c74e3ab46669a3bf0ded8825967 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:33.830374 5742 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08267c74e3ab46669a3bf0ded8825967 [term 1 LEADER]: Becoming Leader. State: Replica: 08267c74e3ab46669a3bf0ded8825967, State: Running, Role: LEADER
I20260926 03:06:33.830962 5742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08267c74e3ab46669a3bf0ded8825967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08267c74e3ab46669a3bf0ded8825967" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } }
I20260926 03:06:33.831336 5737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08267c74e3ab46669a3bf0ded8825967 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:06:33.833762 5744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08267c74e3ab46669a3bf0ded8825967 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08267c74e3ab46669a3bf0ded8825967. Latest consensus state: current_term: 1 leader_uuid: "08267c74e3ab46669a3bf0ded8825967" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08267c74e3ab46669a3bf0ded8825967" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } }
I20260926 03:06:33.833796 5743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08267c74e3ab46669a3bf0ded8825967 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08267c74e3ab46669a3bf0ded8825967" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08267c74e3ab46669a3bf0ded8825967" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37425 } } }
I20260926 03:06:33.834290 5744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08267c74e3ab46669a3bf0ded8825967 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:33.834308 5743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08267c74e3ab46669a3bf0ded8825967 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:33.837510 5751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:06:33.842653 5751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:06:33.851123 5751 catalog_manager.cc:1347] Generated new cluster ID: 454b8dbbe3774ff2bf83e42edef3869e
I20260926 03:06:33.851251 5751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:06:33.861644 5751 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:06:33.863499 5751 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:06:33.875150 5751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08267c74e3ab46669a3bf0ded8825967: Generated new TSK 0
I20260926 03:06:33.876295 5751 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 03:06:33.910146 5767 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 03:06:33.910259 5767 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 03:06:33.911556 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:33.890906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56678 (local address 127.0.97.62:37425)
0926 03:06:33.891111 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:33.891119 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:33.891521 (+ 402us) server_negotiation.cc:407] Connection header received
0926 03:06:33.891765 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:33.891774 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:33.891867 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:33.892052 (+ 185us) server_negotiation.cc:226] Negotiated authn=JWT
0926 03:06:33.892500 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:33.893597 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:33.895065 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:33.895722 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:33.895861 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:33.896431 (+ 570us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 03:06:33.910872 (+ 14441us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46855/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 03:06:33.911273 (+ 401us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56678: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46855/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 03:06:33.911631 5766 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:06:33.890795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37425 (local address 127.0.0.1:56678)
0926 03:06:33.891416 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:33.891432 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:33.891558 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:33.892051 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:33.892055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:33.892064 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 03:06:33.892342 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:33.892351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:33.893745 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:33.893749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:33.894921 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:33.894934 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:33.895880 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:33.895954 (+ 74us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:33.896093 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:33.896130 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 03:06:33.911125 (+ 14995us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46855/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 03:06:33.911353 (+ 228us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:37425: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46855/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":323,"threads_started":1}
W20260926 03:06:33.911684 5740 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56678: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46855/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260926 03:06:33.912953 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 5671
I20260926 03:06:33.914085 5732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:34.005791 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 5671
2026-09-26T03:06:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (708 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 03:06:34.062865 388 mini_oidc.cc:152] Starting JWKS server
I20260926 03:06:34.064774 388 webserver.cc:460] Webserver started at https://127.0.0.1:46017/ using document root <none> and password file <none>
I20260926 03:06:34.064985 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 03:06:34.065588 388 webserver.cc:460] Webserver started at http://127.0.0.1:45905/ using document root <none> and password file <none>
2026-09-26T03:06:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:06:34Z Disabled control of system clock
I20260926 03:06:34.091774 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43517
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43517
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46017/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 03:06:34.411751 5774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:34.412077 5774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:34.412209 5774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:34.419605 5774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:06:34.419742 5774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:34.419798 5774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:34.419874 5774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:06:34.419930 5774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 03:06:34.420019 5774 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 03:06:34.420069 5774 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46017/jwks.json
I20260926 03:06:34.430425 5774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43517
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46017/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:34.431900 5774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:34.433933 5774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:34.446166 5779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:34.446110 5780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:34.446082 5782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:34.451491 5774 server_base.cc:1034] running on GCE node
I20260926 03:06:34.452378 5774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:34.454028 5774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:34.455256 5774 hybrid_clock.cc:648] HybridClock initialized: now 1790391994455197 us; error 53 us; skew 500 ppm
I20260926 03:06:34.455612 5774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:06:34.464805 5774 webserver.cc:460] Webserver started at http://127.0.97.62:39037/ using document root <none> and password file <none>
I20260926 03:06:34.465581 5774 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:34.465721 5774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:34.466082 5774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:34.469276 5774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "64395c324c304b1d9991fabb2e8a9e9d"
format_stamp: "Formatted at 2026-09-26 03:06:34 on dist-test-slave-hmbw"
server_key: "83bde67c3980a009ba27d7215641228c"
server_key_iv: "aee545178006875d857b5e1196ac02c1"
server_key_version: "encryptionkey@0"
I20260926 03:06:34.470077 5774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "64395c324c304b1d9991fabb2e8a9e9d"
format_stamp: "Formatted at 2026-09-26 03:06:34 on dist-test-slave-hmbw"
server_key: "83bde67c3980a009ba27d7215641228c"
server_key_iv: "aee545178006875d857b5e1196ac02c1"
server_key_version: "encryptionkey@0"
I20260926 03:06:34.475533 5774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 03:06:34.479337 5788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:34.480909 5774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:06:34.481083 5774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "64395c324c304b1d9991fabb2e8a9e9d"
format_stamp: "Formatted at 2026-09-26 03:06:34 on dist-test-slave-hmbw"
server_key: "83bde67c3980a009ba27d7215641228c"
server_key_iv: "aee545178006875d857b5e1196ac02c1"
server_key_version: "encryptionkey@0"
I20260926 03:06:34.481302 5774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:34.507651 5774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:34.508522 5774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:34.508814 5774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:34.526827 5774 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43517
I20260926 03:06:34.526836 5839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43517 every 8 connection(s)
I20260926 03:06:34.528213 5774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:06:34.531442 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 5774
I20260926 03:06:34.531646 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:06:34.532254 388 external_mini_cluster.cc:1428] Setting key a997cc5613aa8a23900dfd0b7c6b08a6
I20260926 03:06:34.532419 5840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:34.547780 5840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64395c324c304b1d9991fabb2e8a9e9d: Bootstrap starting.
I20260926 03:06:34.547830 5843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:34.534225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58152 (local address 127.0.97.62:43517)
0926 03:06:34.535427 (+ 1202us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:34.535444 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:34.535511 (+ 67us) server_negotiation.cc:407] Connection header received
0926 03:06:34.537446 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:34.537483 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:34.538203 (+ 720us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:34.538771 (+ 568us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:34.539377 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:34.541368 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:34.542702 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:34.543726 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:34.543911 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:34.544936 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:34.544946 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:34.544967 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:06:34.545714 (+ 747us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:34.545810 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:06:34.545837 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:34.545956 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:34.546146 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 03:06:34.546527 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":378,"threads_started":1}
I20260926 03:06:34.551180 5840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64395c324c304b1d9991fabb2e8a9e9d: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:34.551752 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:06:34.552237 5840 log.cc:824] T 00000000000000000000000000000000 P 64395c324c304b1d9991fabb2e8a9e9d: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:34.555339 5840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64395c324c304b1d9991fabb2e8a9e9d: No bootstrap required, opened a new log
I20260926 03:06:34.563040 5840 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64395c324c304b1d9991fabb2e8a9e9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64395c324c304b1d9991fabb2e8a9e9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43517 } }
I20260926 03:06:34.563491 5840 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64395c324c304b1d9991fabb2e8a9e9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:34.563570 5840 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64395c324c304b1d9991fabb2e8a9e9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64395c324c304b1d9991fabb2e8a9e9d, State: Initialized, Role: FOLLOWER
I20260926 03:06:34.564078 5840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64395c324c304b1d9991fabb2e8a9e9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64395c324c304b1d9991fabb2e8a9e9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43517 } }
I20260926 03:06:34.564268 5840 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64395c324c304b1d9991fabb2e8a9e9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:06:34.564364 5840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64395c324c304b1d9991fabb2e8a9e9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:06:34.564483 5840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64395c324c304b1d9991fabb2e8a9e9d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:34.566623 5840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64395c324c304b1d9991fabb2e8a9e9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64395c324c304b1d9991fabb2e8a9e9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43517 } }
I20260926 03:06:34.567117 5840 leader_election.cc:302] T 00000000000000000000000000000000 P 64395c324c304b1d9991fabb2e8a9e9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64395c324c304b1d9991fabb2e8a9e9d; no voters:
I20260926 03:06:34.567775 5840 leader_election.cc:288] T 00000000000000000000000000000000 P 64395c324c304b1d9991fabb2e8a9e9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:06:34.567979 5845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64395c324c304b1d9991fabb2e8a9e9d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:34.568653 5845 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64395c324c304b1d9991fabb2e8a9e9d [term 1 LEADER]: Becoming Leader. State: Replica: 64395c324c304b1d9991fabb2e8a9e9d, State: Running, Role: LEADER
I20260926 03:06:34.569192 5845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64395c324c304b1d9991fabb2e8a9e9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64395c324c304b1d9991fabb2e8a9e9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43517 } }
I20260926 03:06:34.569837 5840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64395c324c304b1d9991fabb2e8a9e9d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:06:34.571673 5846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64395c324c304b1d9991fabb2e8a9e9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64395c324c304b1d9991fabb2e8a9e9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64395c324c304b1d9991fabb2e8a9e9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43517 } } }
I20260926 03:06:34.572036 5846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64395c324c304b1d9991fabb2e8a9e9d [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:34.572535 5847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64395c324c304b1d9991fabb2e8a9e9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64395c324c304b1d9991fabb2e8a9e9d. Latest consensus state: current_term: 1 leader_uuid: "64395c324c304b1d9991fabb2e8a9e9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64395c324c304b1d9991fabb2e8a9e9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43517 } } }
I20260926 03:06:34.572849 5847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64395c324c304b1d9991fabb2e8a9e9d [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:34.572952 5850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:06:34.578444 5850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:06:34.588109 5850 catalog_manager.cc:1347] Generated new cluster ID: 24eceb8446594fd896ed7f9d04e64aba
I20260926 03:06:34.588228 5850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:06:34.627398 5850 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:06:34.628736 5850 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:06:34.637027 5850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64395c324c304b1d9991fabb2e8a9e9d: Generated new TSK 0
I20260926 03:06:34.637899 5850 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:06:34.664721 5843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:34.655958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58164 (local address 127.0.97.62:43517)
0926 03:06:34.656177 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:34.656184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:34.656655 (+ 471us) server_negotiation.cc:407] Connection header received
0926 03:06:34.656972 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:34.656976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:34.657062 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:34.657227 (+ 165us) server_negotiation.cc:226] Negotiated authn=JWT
0926 03:06:34.657756 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:34.658533 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:34.660580 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:34.661077 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:34.661175 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:34.661834 (+ 659us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 03:06:34.664090 (+ 2256us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46017/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 03:06:34.664425 (+ 335us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58164: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46017/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 03:06:34.664839 5843 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58164: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46017/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 03:06:34.666886 5869 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:06:34.655882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43517 (local address 127.0.0.1:58164)
0926 03:06:34.656526 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:34.656547 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:34.656725 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:34.657263 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:34.657268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:34.657282 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0926 03:06:34.657592 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:34.657604 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:34.658678 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:34.658684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:34.660412 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:34.660429 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:34.661211 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:34.661286 (+ 75us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:34.661488 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:34.661538 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 03:06:34.664314 (+ 2776us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46017/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 03:06:34.666408 (+ 2094us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:43517: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46017/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":1781,"thread_start_us":343,"threads_started":1}
I20260926 03:06:34.668160 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 5774
I20260926 03:06:34.669353 5835 generic_service.cc:196] Checking for leaks (request via RPC)
E20260926 03:06:34.676576 5870 webserver.cc:533] Webserver: sslize failed:
E20260926 03:06:34.676698 5870 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 03:06:34.761734 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 5774
2026-09-26T03:06:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (750 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T03:06:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:06:34Z Disabled control of system clock
I20260926 03:06:34.838477 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36317
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36317
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391950058021-388-0/testchainca.cert with env {}
W20260926 03:06:35.161831 5875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:35.162147 5875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:35.162276 5875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:35.169762 5875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:06:35.169888 5875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:35.169925 5875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:35.170001 5875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:06:35.170058 5875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 03:06:35.170142 5875 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 03:06:35.170200 5875 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 03:06:35.180470 5875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36317
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391950058021-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:35.181833 5875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:35.183164 5875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:35.198781 5880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:35.198824 5883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:35.198870 5881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:35.200013 5875 server_base.cc:1034] running on GCE node
I20260926 03:06:35.200881 5875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:35.202338 5875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:35.203547 5875 hybrid_clock.cc:648] HybridClock initialized: now 1790391995203491 us; error 47 us; skew 500 ppm
I20260926 03:06:35.203945 5875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:06:35.206521 5875 webserver.cc:460] Webserver started at http://127.0.97.62:35245/ using document root <none> and password file <none>
I20260926 03:06:35.207254 5875 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:35.207361 5875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:35.207670 5875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:35.210708 5875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "b74135987532483594a39075bd4230b7"
format_stamp: "Formatted at 2026-09-26 03:06:35 on dist-test-slave-hmbw"
server_key: "9221a35dc1b63027f683b46418968f13"
server_key_iv: "8491e63e5345db9a4024957bd8ac1c37"
server_key_version: "encryptionkey@0"
I20260926 03:06:35.211549 5875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "b74135987532483594a39075bd4230b7"
format_stamp: "Formatted at 2026-09-26 03:06:35 on dist-test-slave-hmbw"
server_key: "9221a35dc1b63027f683b46418968f13"
server_key_iv: "8491e63e5345db9a4024957bd8ac1c37"
server_key_version: "encryptionkey@0"
I20260926 03:06:35.217033 5875 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:06:35.220814 5889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:35.222359 5875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:06:35.222577 5875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "b74135987532483594a39075bd4230b7"
format_stamp: "Formatted at 2026-09-26 03:06:35 on dist-test-slave-hmbw"
server_key: "9221a35dc1b63027f683b46418968f13"
server_key_iv: "8491e63e5345db9a4024957bd8ac1c37"
server_key_version: "encryptionkey@0"
I20260926 03:06:35.222791 5875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:35.246649 5875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:35.247538 5875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:35.247814 5875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:35.265390 5940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36317 every 8 connection(s)
I20260926 03:06:35.265408 5875 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36317
I20260926 03:06:35.266835 5875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:06:35.267179 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 5875
I20260926 03:06:35.267333 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:06:35.268137 388 external_mini_cluster.cc:1428] Setting key b80b8977eb9c1a0ddca99e4e32bca539
I20260926 03:06:35.271072 5941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:35.281239 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:35.269939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52046 (local address 127.0.97.62:36317)
0926 03:06:35.270924 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:35.270935 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:35.270975 (+ 40us) server_negotiation.cc:407] Connection header received
0926 03:06:35.272532 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:35.272552 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:35.273026 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:35.273410 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:35.273966 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:35.275324 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:35.276580 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:35.277261 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:35.277430 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:35.278741 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:35.278750 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:35.278769 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:06:35.279428 (+ 659us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:35.279513 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:06:35.279537 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:35.279654 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:35.279966 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 03:06:35.280297 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":323,"threads_started":1}
I20260926 03:06:35.281878 5941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b74135987532483594a39075bd4230b7: Bootstrap starting.
I20260926 03:06:35.285175 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:06:35.285452 5941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b74135987532483594a39075bd4230b7: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:35.286374 5941 log.cc:824] T 00000000000000000000000000000000 P b74135987532483594a39075bd4230b7: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:35.289371 5941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b74135987532483594a39075bd4230b7: No bootstrap required, opened a new log
I20260926 03:06:35.296689 5941 raft_consensus.cc:348] T 00000000000000000000000000000000 P b74135987532483594a39075bd4230b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b74135987532483594a39075bd4230b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36317 } }
I20260926 03:06:35.297082 5941 raft_consensus.cc:374] T 00000000000000000000000000000000 P b74135987532483594a39075bd4230b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:35.297175 5941 raft_consensus.cc:729] T 00000000000000000000000000000000 P b74135987532483594a39075bd4230b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b74135987532483594a39075bd4230b7, State: Initialized, Role: FOLLOWER
I20260926 03:06:35.297784 5941 consensus_queue.cc:260] T 00000000000000000000000000000000 P b74135987532483594a39075bd4230b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b74135987532483594a39075bd4230b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36317 } }
I20260926 03:06:35.298053 5941 raft_consensus.cc:388] T 00000000000000000000000000000000 P b74135987532483594a39075bd4230b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:06:35.298188 5941 raft_consensus.cc:482] T 00000000000000000000000000000000 P b74135987532483594a39075bd4230b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:06:35.298369 5941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b74135987532483594a39075bd4230b7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:35.300518 5941 raft_consensus.cc:504] T 00000000000000000000000000000000 P b74135987532483594a39075bd4230b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b74135987532483594a39075bd4230b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36317 } }
I20260926 03:06:35.301033 5941 leader_election.cc:302] T 00000000000000000000000000000000 P b74135987532483594a39075bd4230b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b74135987532483594a39075bd4230b7; no voters:
I20260926 03:06:35.301579 5941 leader_election.cc:288] T 00000000000000000000000000000000 P b74135987532483594a39075bd4230b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:06:35.301725 5946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b74135987532483594a39075bd4230b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:35.302645 5946 raft_consensus.cc:686] T 00000000000000000000000000000000 P b74135987532483594a39075bd4230b7 [term 1 LEADER]: Becoming Leader. State: Replica: b74135987532483594a39075bd4230b7, State: Running, Role: LEADER
I20260926 03:06:35.303205 5946 consensus_queue.cc:237] T 00000000000000000000000000000000 P b74135987532483594a39075bd4230b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b74135987532483594a39075bd4230b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36317 } }
I20260926 03:06:35.303524 5941 sys_catalog.cc:564] T 00000000000000000000000000000000 P b74135987532483594a39075bd4230b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:06:35.305933 5948 sys_catalog.cc:455] T 00000000000000000000000000000000 P b74135987532483594a39075bd4230b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b74135987532483594a39075bd4230b7. Latest consensus state: current_term: 1 leader_uuid: "b74135987532483594a39075bd4230b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b74135987532483594a39075bd4230b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36317 } } }
I20260926 03:06:35.306628 5948 sys_catalog.cc:458] T 00000000000000000000000000000000 P b74135987532483594a39075bd4230b7 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:35.306654 5947 sys_catalog.cc:455] T 00000000000000000000000000000000 P b74135987532483594a39075bd4230b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b74135987532483594a39075bd4230b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b74135987532483594a39075bd4230b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36317 } } }
I20260926 03:06:35.307227 5947 sys_catalog.cc:458] T 00000000000000000000000000000000 P b74135987532483594a39075bd4230b7 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:35.308912 5955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:06:35.313459 5955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:06:35.321887 5955 catalog_manager.cc:1347] Generated new cluster ID: 699e882516b44986b680f43573283509
I20260926 03:06:35.322006 5955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:06:35.331825 5955 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:06:35.333133 5955 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:06:35.343323 5955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b74135987532483594a39075bd4230b7: Generated new TSK 0
I20260926 03:06:35.344218 5955 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:06:35.442435 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:35.361089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52056 (local address 127.0.97.62:36317)
0926 03:06:35.361251 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:35.361255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:35.361542 (+ 287us) server_negotiation.cc:407] Connection header received
0926 03:06:35.361766 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:35.361772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:35.361871 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:35.362051 (+ 180us) server_negotiation.cc:226] Negotiated authn=JWT
0926 03:06:35.362445 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:35.363580 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:35.365121 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:35.365831 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:35.365964 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:35.366545 (+ 581us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 03:06:35.441589 (+ 75044us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 03:06:35.442068 (+ 479us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52056: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 03:06:35.442468 5970 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:06:35.360915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36317 (local address 127.0.0.1:52056)
0926 03:06:35.361440 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:35.361456 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:35.361578 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:35.362053 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:35.362057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:35.362066 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 03:06:35.362296 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:35.362305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:35.363805 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:35.363810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:35.364962 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:35.364976 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:35.365986 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:35.366037 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:35.366194 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:35.366230 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 03:06:35.441934 (+ 75704us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 03:06:35.442172 (+ 238us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:36317: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":286,"threads_started":1}
W20260926 03:06:35.442595 5944 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52056: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260926 03:06:35.444146 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 5875
I20260926 03:06:35.445277 5936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:35.543939 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 5875
2026-09-26T03:06:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (762 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (399 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (401 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:06:36 dist-test-slave-hmbw krb5kdc[5985](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:06:36 dist-test-slave-hmbw krb5kdc[5985](info): set up 2 sockets
Sep 26 03:06:36 dist-test-slave-hmbw krb5kdc[5985](info): commencing operation
krb5kdc: starting...
W20260926 03:06:37.351079 388 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:06:37 dist-test-slave-hmbw krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:06:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:06:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:06:37.551414 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36955
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:37.868952 5998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:37.869277 5998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:37.869397 5998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:37.876683 5998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:06:37.876816 5998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:37.876880 5998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:37.876952 5998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:06:37.877008 5998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:06:37.886746 5998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:37.888118 5998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:37.889410 5998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:37.900664 6003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:37.900779 6006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:37.900722 6004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:37.901391 5998 server_base.cc:1034] running on GCE node
I20260926 03:06:37.902091 5998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:37.903445 5998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:37.904649 5998 hybrid_clock.cc:648] HybridClock initialized: now 1790391997904592 us; error 45 us; skew 500 ppm
I20260926 03:06:37.905028 5998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:37 dist-test-slave-hmbw krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:37.908159 5998 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:06:37.909655 5998 webserver.cc:460] Webserver started at http://127.0.97.62:37687/ using document root <none> and password file <none>
I20260926 03:06:37.910343 5998 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:37.910460 5998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:37.910779 5998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:37.913650 5998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "0c6053735b9a464b99ed76bb347dcd42"
format_stamp: "Formatted at 2026-09-26 03:06:37 on dist-test-slave-hmbw"
server_key: "5904b47960fff151badbc776487e70fa"
server_key_iv: "8791dc8541d171f6148d7b44c44344a7"
server_key_version: "encryptionkey@0"
I20260926 03:06:37.914414 5998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "0c6053735b9a464b99ed76bb347dcd42"
format_stamp: "Formatted at 2026-09-26 03:06:37 on dist-test-slave-hmbw"
server_key: "5904b47960fff151badbc776487e70fa"
server_key_iv: "8791dc8541d171f6148d7b44c44344a7"
server_key_version: "encryptionkey@0"
I20260926 03:06:37.919504 5998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:06:37.922958 6013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:37.924255 5998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:06:37.924412 5998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "0c6053735b9a464b99ed76bb347dcd42"
format_stamp: "Formatted at 2026-09-26 03:06:37 on dist-test-slave-hmbw"
server_key: "5904b47960fff151badbc776487e70fa"
server_key_iv: "8791dc8541d171f6148d7b44c44344a7"
server_key_version: "encryptionkey@0"
I20260926 03:06:37.924593 5998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:37.940121 5998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:37.944453 5998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:37.944715 5998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:37.961634 5998 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36955
I20260926 03:06:37.961670 6064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36955 every 8 connection(s)
I20260926 03:06:37.963093 5998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:06:37.966856 6065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:37.969980 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 5998
I20260926 03:06:37.970176 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:06:37.970778 388 external_mini_cluster.cc:1428] Setting key 732e9e534ad5db7b90f1ed5c62545ad0
I20260926 03:06:37.979924 6065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c6053735b9a464b99ed76bb347dcd42: Bootstrap starting.
I20260926 03:06:37.983556 6065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c6053735b9a464b99ed76bb347dcd42: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:37.984557 6065 log.cc:824] T 00000000000000000000000000000000 P 0c6053735b9a464b99ed76bb347dcd42: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:06:37 dist-test-slave-hmbw krb5kdc[5985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:37.987762 6065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c6053735b9a464b99ed76bb347dcd42: No bootstrap required, opened a new log
I20260926 03:06:37.994678 6068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:37.972755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53000 (local address 127.0.97.62:36955)
0926 03:06:37.973549 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:37.973565 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:37.973613 (+ 48us) server_negotiation.cc:407] Connection header received
0926 03:06:37.975546 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:37.975577 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:37.976145 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:37.976661 (+ 516us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:37.977946 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:37.979886 (+ 1940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:37.981274 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:37.982343 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:37.982557 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:37.986857 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:37.986866 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:37.986883 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:37.986926 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:37.990198 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:37.991138 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:37.991148 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:37.991155 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:37.991340 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:37.991796 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:37.991799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:37.991801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:37.992368 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:37.992597 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:37.993048 (+ 451us) server_negotiation.cc:299] Negotiation successful
0926 03:06:37.993428 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":343,"threads_started":1}
I20260926 03:06:37.996397 6065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c6053735b9a464b99ed76bb347dcd42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6053735b9a464b99ed76bb347dcd42" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36955 } }
I20260926 03:06:37.996779 6065 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c6053735b9a464b99ed76bb347dcd42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:37.996932 6065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c6053735b9a464b99ed76bb347dcd42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c6053735b9a464b99ed76bb347dcd42, State: Initialized, Role: FOLLOWER
I20260926 03:06:37.997596 6065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c6053735b9a464b99ed76bb347dcd42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6053735b9a464b99ed76bb347dcd42" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36955 } }
I20260926 03:06:37.997798 6065 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c6053735b9a464b99ed76bb347dcd42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:06:37.997900 6065 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c6053735b9a464b99ed76bb347dcd42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:06:37.998044 6065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c6053735b9a464b99ed76bb347dcd42 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:38.000161 6065 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c6053735b9a464b99ed76bb347dcd42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6053735b9a464b99ed76bb347dcd42" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36955 } }
I20260926 03:06:38.000667 6065 leader_election.cc:302] T 00000000000000000000000000000000 P 0c6053735b9a464b99ed76bb347dcd42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c6053735b9a464b99ed76bb347dcd42; no voters:
I20260926 03:06:38.001183 6065 leader_election.cc:288] T 00000000000000000000000000000000 P 0c6053735b9a464b99ed76bb347dcd42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:06:38.001298 6070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c6053735b9a464b99ed76bb347dcd42 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:38.001890 6070 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c6053735b9a464b99ed76bb347dcd42 [term 1 LEADER]: Becoming Leader. State: Replica: 0c6053735b9a464b99ed76bb347dcd42, State: Running, Role: LEADER
I20260926 03:06:38.002349 6070 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c6053735b9a464b99ed76bb347dcd42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6053735b9a464b99ed76bb347dcd42" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36955 } }
I20260926 03:06:38.003104 6065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c6053735b9a464b99ed76bb347dcd42 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:06:38.005187 6071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c6053735b9a464b99ed76bb347dcd42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c6053735b9a464b99ed76bb347dcd42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6053735b9a464b99ed76bb347dcd42" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36955 } } }
I20260926 03:06:38.005201 6072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c6053735b9a464b99ed76bb347dcd42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c6053735b9a464b99ed76bb347dcd42. Latest consensus state: current_term: 1 leader_uuid: "0c6053735b9a464b99ed76bb347dcd42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6053735b9a464b99ed76bb347dcd42" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36955 } } }
I20260926 03:06:38.005870 6072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c6053735b9a464b99ed76bb347dcd42 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:38.006876 6077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:06:38.006877 6071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c6053735b9a464b99ed76bb347dcd42 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:38.017541 6077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:06:38.028013 6077 catalog_manager.cc:1347] Generated new cluster ID: e1c297b022d74312b2d97f9b0f27ccd1
I20260926 03:06:38.028136 6077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:06:38.047889 6077 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:06:38.049253 6077 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:06:38.069062 6077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c6053735b9a464b99ed76bb347dcd42: Generated new TSK 0
I20260926 03:06:38.070004 6077 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:06:38.126056 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36955
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:38.449282 6091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:38.449589 6091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:38.449717 6091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:38.457075 6091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:38.457218 6091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:38.457373 6091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:06:38.467278 6091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:38.468603 6091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:38.469936 6091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:38.483045 6096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:38.483103 6099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:38.483124 6097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:38.483673 6091 server_base.cc:1034] running on GCE node
I20260926 03:06:38.484279 6091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:38.485191 6091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:38.486402 6091 hybrid_clock.cc:648] HybridClock initialized: now 1790391998486343 us; error 48 us; skew 500 ppm
I20260926 03:06:38.486728 6091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:38 dist-test-slave-hmbw krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:38.489825 6091 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:06:38.491228 6091 webserver.cc:460] Webserver started at http://127.0.97.1:35111/ using document root <none> and password file <none>
I20260926 03:06:38.491899 6091 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:38.492012 6091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:38.492292 6091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:38.495669 6091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-0/data/instance:
uuid: "44921165a385462a9176fadadc86800f"
format_stamp: "Formatted at 2026-09-26 03:06:38 on dist-test-slave-hmbw"
server_key: "e0be3c7afed5d2b846d9553fc5e37dc4"
server_key_iv: "57553a2a9c17f2e58912ba56e6d9c661"
server_key_version: "encryptionkey@0"
I20260926 03:06:38.496495 6091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-0/wal/instance:
uuid: "44921165a385462a9176fadadc86800f"
format_stamp: "Formatted at 2026-09-26 03:06:38 on dist-test-slave-hmbw"
server_key: "e0be3c7afed5d2b846d9553fc5e37dc4"
server_key_iv: "57553a2a9c17f2e58912ba56e6d9c661"
server_key_version: "encryptionkey@0"
I20260926 03:06:38.501910 6091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:06:38.505463 6106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:38.507098 6091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:06:38.507318 6091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "44921165a385462a9176fadadc86800f"
format_stamp: "Formatted at 2026-09-26 03:06:38 on dist-test-slave-hmbw"
server_key: "e0be3c7afed5d2b846d9553fc5e37dc4"
server_key_iv: "57553a2a9c17f2e58912ba56e6d9c661"
server_key_version: "encryptionkey@0"
I20260926 03:06:38.507507 6091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:38.524350 6091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:38.528753 6091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:38.529024 6091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:38.530031 6091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:06:38.531423 6091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:38.531504 6091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:38.531598 6091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:38.531639 6091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:38.560943 6091 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33111
I20260926 03:06:38.560995 6218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33111 every 8 connection(s)
I20260926 03:06:38.562224 6091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:06:38.567147 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 6091
I20260926 03:06:38.567319 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:06:38.567965 388 external_mini_cluster.cc:1428] Setting key ca941650d4fff8926cf37f15efc957ee
Sep 26 03:06:38 dist-test-slave-hmbw krb5kdc[5985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:38.583048 6221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:38.564743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49761 (local address 127.0.97.62:36955)
0926 03:06:38.565145 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:38.565149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:38.566105 (+ 956us) server_negotiation.cc:407] Connection header received
0926 03:06:38.567936 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:38.567943 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:38.568046 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:38.568170 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:38.569851 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:38.570671 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:38.572983 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:38.573381 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:38.573479 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:38.577403 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:38.577409 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:38.577412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:38.577445 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:38.579998 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:38.580890 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:38.580894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:38.580896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:38.581013 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:38.581448 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:38.581451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:38.581453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:38.581701 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:38.581812 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:38.582637 (+ 825us) server_negotiation.cc:299] Negotiation successful
0926 03:06:38.582808 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":194,"threads_started":1}
I20260926 03:06:38.584465 6222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:38.565292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36955 (local address 127.0.97.1:49761)
0926 03:06:38.565959 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:38.565993 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:38.567650 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:38.568364 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:38.568372 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:38.568933 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:38.569647 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:38.569675 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:38.570924 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:38.570930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:38.572812 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:38.572839 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:38.573591 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:38.573597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:38.573886 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:38.574614 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:38.574642 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:38.577173 (+ 2531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:38.580162 (+ 2989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:38.580172 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:38.580191 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:38.580759 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:38.581110 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:38.581114 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:38.581118 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:38.581355 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:38.581817 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:38.581825 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:38.582261 (+ 436us) client_negotiation.cc:769] Sending connection context
0926 03:06:38.582560 (+ 299us) client_negotiation.cc:240] Negotiation successful
0926 03:06:38.583112 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":274,"threads_started":1}
I20260926 03:06:38.586076 6219 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36955
I20260926 03:06:38.586375 6219 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:38.587112 6219 heartbeater.cc:511] Master 127.0.97.62:36955 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:06:38.589305 6030 ts_manager.cc:194] Registered new tserver with Master: 44921165a385462a9176fadadc86800f (127.0.97.1:33111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 03:06:38.591003 6030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:06:38.613895 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36955
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:38.928009 6225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:38.928287 6225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:38.928416 6225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:38.935726 6225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:38.935840 6225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:38.935982 6225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:06:38.945793 6225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:38.947212 6225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:38.948568 6225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:38.961907 6230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:38.961931 6231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:38.962845 6225 server_base.cc:1034] running on GCE node
W20260926 03:06:38.966809 6233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:38.967557 6225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:38.968623 6225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:38.969900 6225 hybrid_clock.cc:648] HybridClock initialized: now 1790391998969846 us; error 44 us; skew 500 ppm
I20260926 03:06:38.970221 6225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:38 dist-test-slave-hmbw krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:38.973512 6225 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:06:38.975080 6225 webserver.cc:460] Webserver started at http://127.0.97.2:39183/ using document root <none> and password file <none>
I20260926 03:06:38.975728 6225 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:38.975919 6225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:38.976258 6225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:38.979307 6225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-1/data/instance:
uuid: "777ebb79616340c1a9831c850596c734"
format_stamp: "Formatted at 2026-09-26 03:06:38 on dist-test-slave-hmbw"
server_key: "7bf771f43e1836cb2dabf66d153007ba"
server_key_iv: "4a854e4e6d9804ca91e1079b902b0b0b"
server_key_version: "encryptionkey@0"
I20260926 03:06:38.980051 6225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-1/wal/instance:
uuid: "777ebb79616340c1a9831c850596c734"
format_stamp: "Formatted at 2026-09-26 03:06:38 on dist-test-slave-hmbw"
server_key: "7bf771f43e1836cb2dabf66d153007ba"
server_key_iv: "4a854e4e6d9804ca91e1079b902b0b0b"
server_key_version: "encryptionkey@0"
I20260926 03:06:38.985355 6225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:06:38.989058 6240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:38.990614 6225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 03:06:38.990777 6225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "777ebb79616340c1a9831c850596c734"
format_stamp: "Formatted at 2026-09-26 03:06:38 on dist-test-slave-hmbw"
server_key: "7bf771f43e1836cb2dabf66d153007ba"
server_key_iv: "4a854e4e6d9804ca91e1079b902b0b0b"
server_key_version: "encryptionkey@0"
I20260926 03:06:38.990996 6225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:39.009490 6225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:39.014287 6225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:39.014621 6225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:39.015659 6225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:06:39.017061 6225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:39.017134 6225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:39.017248 6225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:39.017297 6225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:39.050062 6225 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39273
I20260926 03:06:39.050105 6352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39273 every 8 connection(s)
I20260926 03:06:39.051406 6225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:06:39.052587 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 6225
I20260926 03:06:39.052749 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:06:39.053300 388 external_mini_cluster.cc:1428] Setting key 51dd5bde14321ce10781dc473f1a2d90
Sep 26 03:06:39 dist-test-slave-hmbw krb5kdc[5985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 03:06:39.074671 6221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:39.054285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33681 (local address 127.0.97.62:36955)
0926 03:06:39.054461 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:39.054467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:39.055966 (+ 1499us) server_negotiation.cc:407] Connection header received
0926 03:06:39.057887 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:39.057892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:39.057991 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:39.058096 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:39.059753 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:39.060658 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:39.063272 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:39.063687 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:39.063847 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:39.067836 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:39.067842 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:39.067847 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:39.067891 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:39.071359 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:39.072209 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:39.072215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:39.072219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:39.072366 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:39.072822 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:39.072827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:39.072830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:39.073178 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:39.073333 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:39.074149 (+ 816us) server_negotiation.cc:299] Negotiation successful
0926 03:06:39.074367 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:06:39.075732 6355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:39.054892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36955 (local address 127.0.97.2:33681)
0926 03:06:39.055791 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:39.055840 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:39.057586 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:39.058380 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:39.058393 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:39.058982 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:39.059562 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:39.059579 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:39.060887 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:39.060893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:39.063081 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:39.063098 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:39.063972 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:39.063978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:39.064277 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:39.064990 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:39.065023 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:39.067544 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:39.071551 (+ 4007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:39.071559 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:39.071575 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:39.072057 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:39.072500 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:39.072503 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:39.072506 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:39.072698 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:39.073360 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:39.073369 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:39.073798 (+ 429us) client_negotiation.cc:769] Sending connection context
0926 03:06:39.074051 (+ 253us) client_negotiation.cc:240] Negotiation successful
0926 03:06:39.074547 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":382,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 03:06:39.077288 6353 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36955
I20260926 03:06:39.077600 6353 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:39.078486 6353 heartbeater.cc:511] Master 127.0.97.62:36955 requested a full tablet report, sending...
I20260926 03:06:39.080142 6030 ts_manager.cc:194] Registered new tserver with Master: 777ebb79616340c1a9831c850596c734 (127.0.97.2:39273)
I20260926 03:06:39.081157 6030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:06:39.102293 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36955
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:39.426877 6358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:39.427184 6358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:39.427318 6358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:39.434904 6358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:39.435039 6358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:39.435187 6358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:06:39.445348 6358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:39.446700 6358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:39.448001 6358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:39.461021 6363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:39.460992 6364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:39.461517 6366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:39.461759 6358 server_base.cc:1034] running on GCE node
I20260926 03:06:39.462404 6358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:39.463367 6358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:39.464564 6358 hybrid_clock.cc:648] HybridClock initialized: now 1790391999464519 us; error 44 us; skew 500 ppm
I20260926 03:06:39.464865 6358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:39 dist-test-slave-hmbw krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:39.473279 6358 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:06:39.474785 6358 webserver.cc:460] Webserver started at http://127.0.97.3:37199/ using document root <none> and password file <none>
I20260926 03:06:39.475467 6358 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:39.475586 6358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:39.475878 6358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:39.478972 6358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-2/data/instance:
uuid: "fdf3932af2824a45a5e3b3c6c1bcc669"
format_stamp: "Formatted at 2026-09-26 03:06:39 on dist-test-slave-hmbw"
server_key: "138a7b92de72088ee7cfc06ab3536c7d"
server_key_iv: "80bb83565e7cab5e8234fdd06503e366"
server_key_version: "encryptionkey@0"
I20260926 03:06:39.479703 6358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fdf3932af2824a45a5e3b3c6c1bcc669"
format_stamp: "Formatted at 2026-09-26 03:06:39 on dist-test-slave-hmbw"
server_key: "138a7b92de72088ee7cfc06ab3536c7d"
server_key_iv: "80bb83565e7cab5e8234fdd06503e366"
server_key_version: "encryptionkey@0"
I20260926 03:06:39.484669 6358 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:06:39.488235 6373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:39.489634 6358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:06:39.489779 6358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "fdf3932af2824a45a5e3b3c6c1bcc669"
format_stamp: "Formatted at 2026-09-26 03:06:39 on dist-test-slave-hmbw"
server_key: "138a7b92de72088ee7cfc06ab3536c7d"
server_key_iv: "80bb83565e7cab5e8234fdd06503e366"
server_key_version: "encryptionkey@0"
I20260926 03:06:39.490005 6358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:39.529114 6358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:39.533140 6358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:39.533391 6358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:39.534310 6358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:06:39.535656 6358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:39.535727 6358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:39.535825 6358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:39.535873 6358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:39.565366 6358 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38015
I20260926 03:06:39.565412 6485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38015 every 8 connection(s)
I20260926 03:06:39.566634 6358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:06:39.575070 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 6358
I20260926 03:06:39.575222 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:06:39.575615 388 external_mini_cluster.cc:1428] Setting key 39a051b8f45822a4cde5ea4099794657
Sep 26 03:06:39 dist-test-slave-hmbw krb5kdc[5985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:39.587169 6221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:39.569132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47193 (local address 127.0.97.62:36955)
0926 03:06:39.569294 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:39.569299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:39.570467 (+ 1168us) server_negotiation.cc:407] Connection header received
0926 03:06:39.572343 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:39.572348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:39.572436 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:39.572521 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:39.574076 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:39.574966 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:39.576819 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:39.577409 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:39.577528 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:39.581166 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:39.581171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:39.581175 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:39.581211 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:39.583844 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:39.584627 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:39.584632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:39.584635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:39.584859 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:39.585382 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:39.585386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:39.585389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:39.585772 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:39.585892 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:39.586681 (+ 789us) server_negotiation.cc:299] Negotiation successful
0926 03:06:39.586897 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 03:06:39.588208 6488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:39.569578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36955 (local address 127.0.97.3:47193)
0926 03:06:39.570280 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:39.570322 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:39.572048 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:39.572770 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:39.572779 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:39.573308 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:39.573896 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:39.573913 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:39.575133 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:39.575137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:39.576642 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:39.576675 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:39.577654 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:39.577662 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:39.578012 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:39.578762 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:39.578786 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:39.580926 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:39.584004 (+ 3078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:39.584012 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:39.584028 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:39.584473 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:39.585024 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:39.585028 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:39.585031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:39.585260 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:39.585914 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:39.585921 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:39.586273 (+ 352us) client_negotiation.cc:769] Sending connection context
0926 03:06:39.586566 (+ 293us) client_negotiation.cc:240] Negotiation successful
0926 03:06:39.587052 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":297,"threads_started":1}
I20260926 03:06:39.589773 6486 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36955
I20260926 03:06:39.590106 6486 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:39.591123 6486 heartbeater.cc:511] Master 127.0.97.62:36955 requested a full tablet report, sending...
I20260926 03:06:39.592489 6030 ts_manager.cc:194] Registered new tserver with Master: fdf3932af2824a45a5e3b3c6c1bcc669 (127.0.97.3:38015)
I20260926 03:06:39.593315 6030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47193
I20260926 03:06:39.594612 6219 heartbeater.cc:503] Master 127.0.97.62:36955 was elected leader, sending a full tablet report...
I20260926 03:06:39.604032 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/test-admin.keytab.
Sep 26 03:06:39 dist-test-slave-hmbw krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:39.629782 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 03:06:39.634895 6493 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:06:39.632483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60464 (local address 127.0.97.1:33111)
0926 03:06:39.633112 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:39.633118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:39.633143 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:06:39.633200 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:39.633206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:39.633420 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:39.633555 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:39.634479 (+ 924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60464: BlockingRecv error: recv got EOF from 127.0.0.1:60464 (error 108)
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":309,"threads_started":1}
W20260926 03:06:39.635010 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/krb5cc' of type FILE
Sep 26 03:06:39 dist-test-slave-hmbw krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:39.636798 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 03:06:39 dist-test-slave-hmbw krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:39.638147 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 03:06:39 dist-test-slave-hmbw krb5kdc[5985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:06:39.653375 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:39.639792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60474 (local address 127.0.97.1:33111)
0926 03:06:39.640075 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:39.640081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:39.640291 (+ 210us) server_negotiation.cc:407] Connection header received
0926 03:06:39.640441 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:39.640445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:39.640510 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:39.640596 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:39.641609 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:39.642659 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:39.643842 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:39.644420 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:39.644522 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:39.648084 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:39.648097 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:39.648113 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:39.648151 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:39.650900 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:39.651563 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:39.651570 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:39.651573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:39.651682 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:39.651982 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:39.651986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:39.651988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:39.652316 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:39.652540 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:39.652855 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 03:06:39.653074 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260926 03:06:39.658711 6493 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:06:39.657125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60486 (local address 127.0.97.1:33111)
0926 03:06:39.657273 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:39.657277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:39.657531 (+ 254us) server_negotiation.cc:407] Connection header received
0926 03:06:39.657665 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:39.657669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:39.657742 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:39.657832 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:39.658450 (+ 618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60486: BlockingRecv error: recv got EOF from 127.0.0.1:60486 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 03:06:39.659025 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/krb5cc' of type FILE
Sep 26 03:06:39 dist-test-slave-hmbw krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:39.660488 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 03:06:39 dist-test-slave-hmbw krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:39.661691 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 03:06:39 dist-test-slave-hmbw krb5kdc[5985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:06:39.675432 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:39.663044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60498 (local address 127.0.97.1:33111)
0926 03:06:39.663241 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:39.663247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:39.663425 (+ 178us) server_negotiation.cc:407] Connection header received
0926 03:06:39.663569 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:39.663573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:39.663634 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:39.663711 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:39.664643 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:39.665335 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:39.666444 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:39.666904 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:39.667004 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:39.670371 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:39.670377 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:39.670380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:39.670416 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:39.672964 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:39.673622 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:39.673626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:39.673628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:39.673759 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:39.674119 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:39.674123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:39.674126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:39.674423 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:39.674668 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:39.674956 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 03:06:39.675130 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 03:06:39.679580 6493 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:06:39.677970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60510 (local address 127.0.97.1:33111)
0926 03:06:39.678141 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:39.678146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:39.678225 (+ 79us) server_negotiation.cc:407] Connection header received
0926 03:06:39.678404 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:39.678409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:39.678479 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:39.678589 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:39.679356 (+ 767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60510: BlockingRecv error: recv got EOF from 127.0.0.1:60510 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 03:06:39.679661 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391950058021-388-0/krb5kdc/krb5cc' of type FILE
Sep 26 03:06:39 dist-test-slave-hmbw krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:39.681078 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 03:06:39 dist-test-slave-hmbw krb5kdc[5985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:39.682348 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 03:06:39 dist-test-slave-hmbw krb5kdc[5985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:06:39.696513 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:39.683677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60514 (local address 127.0.97.1:33111)
0926 03:06:39.683895 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:39.683901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:39.684078 (+ 177us) server_negotiation.cc:407] Connection header received
0926 03:06:39.684239 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:39.684242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:39.684297 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:39.684374 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:39.685303 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:39.686006 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:39.687249 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:39.687655 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:39.687762 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:39.691179 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:39.691184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:39.691188 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:39.691222 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:39.693943 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:39.694665 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:39.694669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:39.694672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:39.694882 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:39.695254 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:39.695258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:39.695260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:39.695552 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:39.695775 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:39.696072 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 03:06:39.696240 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 03:06:39.698020 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 6091
I20260926 03:06:39.710647 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:39.698692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60516 (local address 127.0.97.1:33111)
0926 03:06:39.698884 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:39.698888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:39.699206 (+ 318us) server_negotiation.cc:407] Connection header received
0926 03:06:39.699344 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:39.699347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:39.699403 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:39.699472 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:39.700442 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:39.701166 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:39.702289 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:39.702817 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:39.702931 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:39.704919 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:39.704925 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:39.704928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:39.704961 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:39.707688 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:39.708518 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:39.708523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:39.708525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:39.708707 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:39.709139 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:39.709143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:39.709146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:39.709451 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:39.709660 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:39.710119 (+ 459us) server_negotiation.cc:299] Negotiation successful
0926 03:06:39.710319 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 03:06:39.711194 6214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:39.793653 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 6091
I20260926 03:06:39.806875 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 6225
Sep 26 03:06:39 dist-test-slave-hmbw krb5kdc[5985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:06:39.824667 6506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:39.807781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49400 (local address 127.0.97.2:39273)
0926 03:06:39.808386 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:39.808394 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:39.808420 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:06:39.808484 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:39.808491 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:39.808691 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:39.808904 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:39.810244 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:39.811777 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:39.813736 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:39.814685 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:39.814826 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:39.819282 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:39.819299 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:39.819315 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:39.819370 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:39.821978 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:39.822785 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:39.822793 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:39.822795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:39.822943 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:39.823274 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:39.823277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:39.823279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:39.823621 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:39.823851 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:39.824090 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 03:06:39.824301 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":329,"threads_started":1}
I20260926 03:06:39.825981 6348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:39.907488 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 6225
I20260926 03:06:39.920460 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 6358
Sep 26 03:06:39 dist-test-slave-hmbw krb5kdc[5985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:06:39.936935 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:39.921363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53900 (local address 127.0.97.3:38015)
0926 03:06:39.921990 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:39.921997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:39.922024 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:06:39.922080 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:39.922086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:39.922324 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:39.922445 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:39.923498 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:39.924527 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:39.925725 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:39.926283 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:39.926397 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:39.930703 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:39.930718 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:39.930742 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:39.930784 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:39.933876 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:39.934757 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:39.934820 (+ 63us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:39.934823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:39.934975 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:39.935356 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:39.935360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:39.935362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:39.935724 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:39.936024 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:39.936363 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 03:06:39.936588 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":319,"threads_started":1}
I20260926 03:06:39.938354 6481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:40.022774 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 6358
I20260926 03:06:40.036265 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 5998
I20260926 03:06:40.037667 6060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:40.140254 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 5998
2026-09-26T03:06:40Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3800 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:06:40 dist-test-slave-hmbw krb5kdc[6513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:06:40 dist-test-slave-hmbw krb5kdc[6513](info): set up 2 sockets
Sep 26 03:06:40 dist-test-slave-hmbw krb5kdc[6513](info): commencing operation
krb5kdc: starting...
W20260926 03:06:41.166934 388 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:06:41 dist-test-slave-hmbw krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:06:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:06:41Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260926 03:06:41.369608 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36297
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:41.689515 6526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:41.689826 6526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:41.689914 6526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:41.697273 6526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:06:41.697387 6526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:41.697449 6526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:41.697503 6526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:06:41.697546 6526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:06:41.707566 6526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:41.709016 6526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:41.710318 6526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:41.721904 6531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:41.721920 6534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:41.721832 6532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:41.722791 6526 server_base.cc:1034] running on GCE node
I20260926 03:06:41.723590 6526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:41.724992 6526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:41.726203 6526 hybrid_clock.cc:648] HybridClock initialized: now 1790392001726153 us; error 34 us; skew 500 ppm
I20260926 03:06:41.726565 6526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:41 dist-test-slave-hmbw krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392001, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:41.729501 6526 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260926 03:06:41.730993 6526 webserver.cc:460] Webserver started at http://127.0.97.62:34565/ using document root <none> and password file <none>
I20260926 03:06:41.731663 6526 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:41.731776 6526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:41.732069 6526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:41.735020 6526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71"
format_stamp: "Formatted at 2026-09-26 03:06:41 on dist-test-slave-hmbw"
server_key: "8629265d49e93b705f6e67296083670a"
server_key_iv: "e4200f1e2ac8f3e34407704b66ad2b5f"
server_key_version: "encryptionkey@0"
I20260926 03:06:41.735778 6526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71"
format_stamp: "Formatted at 2026-09-26 03:06:41 on dist-test-slave-hmbw"
server_key: "8629265d49e93b705f6e67296083670a"
server_key_iv: "e4200f1e2ac8f3e34407704b66ad2b5f"
server_key_version: "encryptionkey@0"
I20260926 03:06:41.740978 6526 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:06:41.744582 6541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:41.746011 6526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:06:41.746208 6526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71"
format_stamp: "Formatted at 2026-09-26 03:06:41 on dist-test-slave-hmbw"
server_key: "8629265d49e93b705f6e67296083670a"
server_key_iv: "e4200f1e2ac8f3e34407704b66ad2b5f"
server_key_version: "encryptionkey@0"
I20260926 03:06:41.746402 6526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:41.768599 6526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:41.774103 6526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:41.774384 6526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:41.791481 6526 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36297
I20260926 03:06:41.791508 6592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36297 every 8 connection(s)
I20260926 03:06:41.792764 6526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:06:41.796538 6593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:41.798772 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 6526
I20260926 03:06:41.798954 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:06:41.799557 388 external_mini_cluster.cc:1428] Setting key ac030c7763c3115a75444d034aa94d20
I20260926 03:06:41.808691 6593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71: Bootstrap starting.
I20260926 03:06:41.812860 6593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:41.814148 6593 log.cc:824] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:06:41 dist-test-slave-hmbw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:06:41.818174 6593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71: No bootstrap required, opened a new log
I20260926 03:06:41.825279 6596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:41.801645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44178 (local address 127.0.97.62:36297)
0926 03:06:41.802403 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:41.802414 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:41.802455 (+ 41us) server_negotiation.cc:407] Connection header received
0926 03:06:41.804095 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:41.804118 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:41.804539 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:41.804902 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:41.806686 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:41.808355 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:41.810139 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:41.811220 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:41.811412 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:41.816366 (+ 4954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:41.816377 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:41.816399 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:41.816447 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:41.820450 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:41.821253 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:41.821266 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:41.821276 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:41.821497 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:41.821826 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:41.821831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:41.821835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:41.822631 (+ 796us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:41.822914 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:41.823329 (+ 415us) server_negotiation.cc:299] Negotiation successful
0926 03:06:41.823817 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":333,"threads_started":1}
I20260926 03:06:41.828315 6593 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } }
I20260926 03:06:41.828694 6593 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:41.828771 6593 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8e4b0a3ad2449c1be9c81c740b8ef71, State: Initialized, Role: FOLLOWER
I20260926 03:06:41.829283 6593 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } }
I20260926 03:06:41.829475 6593 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:06:41.829569 6593 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:06:41.829696 6593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:41.831818 6593 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } }
I20260926 03:06:41.832333 6593 leader_election.cc:302] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8e4b0a3ad2449c1be9c81c740b8ef71; no voters:
I20260926 03:06:41.832993 6593 leader_election.cc:288] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:06:41.833122 6598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:41.833724 6598 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [term 1 LEADER]: Becoming Leader. State: Replica: d8e4b0a3ad2449c1be9c81c740b8ef71, State: Running, Role: LEADER
I20260926 03:06:41.834231 6598 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } }
I20260926 03:06:41.834998 6593 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:06:41.836885 6599 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } }
I20260926 03:06:41.837332 6599 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:41.836861 6600 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8e4b0a3ad2449c1be9c81c740b8ef71. Latest consensus state: current_term: 1 leader_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } }
I20260926 03:06:41.839110 6600 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:41.840238 6603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:06:41.845818 6603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:06:41.855185 6603 catalog_manager.cc:1347] Generated new cluster ID: 6aeb71c0cc784ff4b8ab8bd34d9dd50f
I20260926 03:06:41.855307 6603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:06:41.869573 6603 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:06:41.871518 6603 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:06:41.883687 6603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71: Generated new TSK 0
I20260926 03:06:41.884750 6603 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260926 03:06:41.942577 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36297
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:42.278646 6619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:42.278935 6619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:42.279052 6619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:42.286412 6619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:42.286568 6619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:42.286720 6619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:06:42.297044 6619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:42.298378 6619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:42.299762 6619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:42.316226 6627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:42.316288 6625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:42.316381 6624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:42.317333 6619 server_base.cc:1034] running on GCE node
I20260926 03:06:42.318317 6619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:42.319891 6619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:42.321166 6619 hybrid_clock.cc:648] HybridClock initialized: now 1790392002321107 us; error 66 us; skew 500 ppm
I20260926 03:06:42.321524 6619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:42 dist-test-slave-hmbw krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392002, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:42.325071 6619 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260926 03:06:42.326680 6619 webserver.cc:460] Webserver started at http://127.0.97.1:45957/ using document root <none> and password file <none>
I20260926 03:06:42.327436 6619 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:42.327539 6619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:42.327805 6619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:42.330749 6619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/data/instance:
uuid: "1f35873792864b84a8ee278ba9b71a9d"
format_stamp: "Formatted at 2026-09-26 03:06:42 on dist-test-slave-hmbw"
server_key: "accc9a5378001f28e4d5d0808c814358"
server_key_iv: "4e9217140e5691d56762fb775cb690df"
server_key_version: "encryptionkey@0"
I20260926 03:06:42.331530 6619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1f35873792864b84a8ee278ba9b71a9d"
format_stamp: "Formatted at 2026-09-26 03:06:42 on dist-test-slave-hmbw"
server_key: "accc9a5378001f28e4d5d0808c814358"
server_key_iv: "4e9217140e5691d56762fb775cb690df"
server_key_version: "encryptionkey@0"
I20260926 03:06:42.336831 6619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:06:42.340374 6634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:42.341840 6619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:06:42.342002 6619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "1f35873792864b84a8ee278ba9b71a9d"
format_stamp: "Formatted at 2026-09-26 03:06:42 on dist-test-slave-hmbw"
server_key: "accc9a5378001f28e4d5d0808c814358"
server_key_iv: "4e9217140e5691d56762fb775cb690df"
server_key_version: "encryptionkey@0"
I20260926 03:06:42.342209 6619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:42.360687 6619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:42.365078 6619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:42.365351 6619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:42.366384 6619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:06:42.367874 6619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:42.367959 6619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:42.368059 6619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:42.368115 6619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:42.397837 6619 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43877
I20260926 03:06:42.397882 6746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43877 every 8 connection(s)
I20260926 03:06:42.399191 6619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:06:42.402333 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 6619
I20260926 03:06:42.402545 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:06:42.403169 388 external_mini_cluster.cc:1428] Setting key 86e6b079522a3502cefffaaaa6ab6972
Sep 26 03:06:42 dist-test-slave-hmbw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392002, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:06:42.420518 6749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:42.401669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58127 (local address 127.0.97.62:36297)
0926 03:06:42.402107 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:42.402114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:42.403244 (+ 1130us) server_negotiation.cc:407] Connection header received
0926 03:06:42.405550 (+ 2306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:42.405555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:42.405638 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:42.405765 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:42.407497 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:42.408416 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:42.410004 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:42.410576 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:42.410693 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:42.414796 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:42.414802 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:42.414805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:42.414843 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:42.417283 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:42.418164 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:42.418168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:42.418171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:42.418284 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:42.418799 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:42.418803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:42.418805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:42.419076 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:42.419185 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:42.420090 (+ 905us) server_negotiation.cc:299] Negotiation successful
0926 03:06:42.420256 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":192,"threads_started":1}
I20260926 03:06:42.421955 6750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:42.402246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.97.1:58127)
0926 03:06:42.403050 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:42.403109 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:42.405233 (+ 2124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:42.406011 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:42.406042 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:42.406667 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:42.407309 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:42.407329 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:42.408579 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:42.408583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:42.409846 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:42.409860 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:42.410772 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:42.410780 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:42.411069 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:42.411773 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:42.411807 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:42.414541 (+ 2734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:42.417443 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:42.417453 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:42.417471 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:42.418046 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:42.418387 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:42.418392 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:42.418396 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:42.418683 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:42.419204 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:42.419212 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:42.419685 (+ 473us) client_negotiation.cc:769] Sending connection context
0926 03:06:42.420005 (+ 320us) client_negotiation.cc:240] Negotiation successful
0926 03:06:42.420566 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":338,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:06:42.423517 6747 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36297
I20260926 03:06:42.423835 6747 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:42.424569 6747 heartbeater.cc:511] Master 127.0.97.62:36297 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260926 03:06:42.426836 6558 ts_manager.cc:194] Registered new tserver with Master: 1f35873792864b84a8ee278ba9b71a9d (127.0.97.1:43877)
I20260926 03:06:42.428414 6558 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260926 03:06:42.451227 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36297
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:42.771338 6753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:42.771662 6753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:42.771759 6753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:42.780445 6753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:42.780583 6753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:42.780706 6753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:06:42.791945 6753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:42.793226 6753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:42.794586 6753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:42.808386 6758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:42.808382 6759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:42.808403 6761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:42.808984 6753 server_base.cc:1034] running on GCE node
I20260926 03:06:42.809638 6753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:42.810653 6753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:42.811884 6753 hybrid_clock.cc:648] HybridClock initialized: now 1790392002811846 us; error 31 us; skew 500 ppm
I20260926 03:06:42.812255 6753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:42 dist-test-slave-hmbw krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392002, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:42.815526 6753 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260926 03:06:42.817183 6753 webserver.cc:460] Webserver started at http://127.0.97.2:43247/ using document root <none> and password file <none>
I20260926 03:06:42.817898 6753 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:42.818022 6753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:42.818328 6753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:42.821334 6753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/data/instance:
uuid: "0c1d74ffdafa4ce78d561501e3672999"
format_stamp: "Formatted at 2026-09-26 03:06:42 on dist-test-slave-hmbw"
server_key: "ce981ada0526d303926a00b631bfc057"
server_key_iv: "6764b2955ceb16a82bb78a0fb3c01fd8"
server_key_version: "encryptionkey@0"
I20260926 03:06:42.822124 6753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0c1d74ffdafa4ce78d561501e3672999"
format_stamp: "Formatted at 2026-09-26 03:06:42 on dist-test-slave-hmbw"
server_key: "ce981ada0526d303926a00b631bfc057"
server_key_iv: "6764b2955ceb16a82bb78a0fb3c01fd8"
server_key_version: "encryptionkey@0"
I20260926 03:06:42.827581 6753 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 03:06:42.831689 6768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:42.833461 6753 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 03:06:42.833611 6753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "0c1d74ffdafa4ce78d561501e3672999"
format_stamp: "Formatted at 2026-09-26 03:06:42 on dist-test-slave-hmbw"
server_key: "ce981ada0526d303926a00b631bfc057"
server_key_iv: "6764b2955ceb16a82bb78a0fb3c01fd8"
server_key_version: "encryptionkey@0"
I20260926 03:06:42.833787 6753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:42.853014 6753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:42.856937 6753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:42.857167 6753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:42.858026 6753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:06:42.859378 6753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:42.859452 6753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:42.859541 6753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:42.859582 6753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:42.889077 6753 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34799
I20260926 03:06:42.889122 6880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34799 every 8 connection(s)
I20260926 03:06:42.890292 6753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:06:42.890731 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 6753
I20260926 03:06:42.890911 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:06:42.891422 388 external_mini_cluster.cc:1428] Setting key e4b230f02f0cf929b8402a9c1b95ea7d
Sep 26 03:06:42 dist-test-slave-hmbw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392002, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:06:42.912360 6749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:42.892954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42529 (local address 127.0.97.62:36297)
0926 03:06:42.893139 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:42.893145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:42.894554 (+ 1409us) server_negotiation.cc:407] Connection header received
0926 03:06:42.896982 (+ 2428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:42.896988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:42.897083 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:42.897173 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:42.899224 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:42.900033 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:42.901795 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:42.902377 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:42.902527 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:42.906087 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:42.906093 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:42.906098 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:42.906137 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:42.909125 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:42.909880 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:42.909885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:42.909889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:42.910039 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:42.910450 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:42.910454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:42.910457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:42.910882 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:42.911009 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:42.911948 (+ 939us) server_negotiation.cc:299] Negotiation successful
0926 03:06:42.912116 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 03:06:42.913578 6883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:42.893538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.97.2:42529)
0926 03:06:42.894376 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:42.894433 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:42.896636 (+ 2203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:42.897458 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:42.897470 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:42.898162 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:42.898987 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:42.899016 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:42.900182 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:42.900188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:42.901643 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:42.901655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:42.902612 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:42.902617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:42.902825 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:42.903449 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:42.903475 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:42.905802 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:42.909283 (+ 3481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:42.909290 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:42.909306 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:42.909742 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:42.910150 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:42.910153 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:42.910156 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:42.910332 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:42.911029 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:42.911038 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:42.911537 (+ 499us) client_negotiation.cc:769] Sending connection context
0926 03:06:42.911859 (+ 322us) client_negotiation.cc:240] Negotiation successful
0926 03:06:42.912414 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":336,"threads_started":1}
I20260926 03:06:42.915135 6881 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36297
I20260926 03:06:42.915402 6881 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
I20260926 03:06:42.916118 6881 heartbeater.cc:511] Master 127.0.97.62:36297 requested a full tablet report, sending...
I20260926 03:06:42.917536 6558 ts_manager.cc:194] Registered new tserver with Master: 0c1d74ffdafa4ce78d561501e3672999 (127.0.97.2:34799)
I20260926 03:06:42.918318 6558 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42529
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260926 03:06:42.939249 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36297
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:43.261729 6886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:43.262035 6886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:43.262153 6886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:43.269793 6886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:43.269913 6886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:43.270059 6886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:06:43.280153 6886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:43.281441 6886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:43.282905 6886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:43.295584 6894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:43.295624 6892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:43.295655 6891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:43.296612 6886 server_base.cc:1034] running on GCE node
I20260926 03:06:43.297251 6886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:43.298123 6886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:43.299351 6886 hybrid_clock.cc:648] HybridClock initialized: now 1790392003299287 us; error 46 us; skew 500 ppm
I20260926 03:06:43.299678 6886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:43 dist-test-slave-hmbw krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392003, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:43.303190 6886 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260926 03:06:43.304764 6886 webserver.cc:460] Webserver started at http://127.0.97.3:42529/ using document root <none> and password file <none>
I20260926 03:06:43.305487 6886 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:43.305593 6886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:43.305879 6886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:43.308976 6886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/data/instance:
uuid: "4065ff7c51bd4a2499e8dd30a4e51c81"
format_stamp: "Formatted at 2026-09-26 03:06:43 on dist-test-slave-hmbw"
server_key: "97f8dba83f8b4f61ffc0e6471865d5c9"
server_key_iv: "850122895d29054f0d850ec2b9d74f18"
server_key_version: "encryptionkey@0"
I20260926 03:06:43.309777 6886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4065ff7c51bd4a2499e8dd30a4e51c81"
format_stamp: "Formatted at 2026-09-26 03:06:43 on dist-test-slave-hmbw"
server_key: "97f8dba83f8b4f61ffc0e6471865d5c9"
server_key_iv: "850122895d29054f0d850ec2b9d74f18"
server_key_version: "encryptionkey@0"
I20260926 03:06:43.315197 6886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 03:06:43.319069 6901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:43.320672 6886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:06:43.320853 6886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "4065ff7c51bd4a2499e8dd30a4e51c81"
format_stamp: "Formatted at 2026-09-26 03:06:43 on dist-test-slave-hmbw"
server_key: "97f8dba83f8b4f61ffc0e6471865d5c9"
server_key_iv: "850122895d29054f0d850ec2b9d74f18"
server_key_version: "encryptionkey@0"
I20260926 03:06:43.321054 6886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:43.362672 6886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:43.367018 6886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:43.367300 6886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:43.368278 6886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:06:43.369670 6886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:43.369741 6886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:43.369838 6886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:43.369892 6886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:43.399785 6886 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44419
I20260926 03:06:43.399817 7013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44419 every 8 connection(s)
I20260926 03:06:43.401103 6886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:06:43.408856 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 6886
I20260926 03:06:43.409011 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:06:43.409432 388 external_mini_cluster.cc:1428] Setting key bdd2f18215a1654bd5eacc6d324fffe3
Sep 26 03:06:43 dist-test-slave-hmbw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392003, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:06:43.421001 6749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:43.403673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58665 (local address 127.0.97.62:36297)
0926 03:06:43.403868 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:43.403875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:43.405204 (+ 1329us) server_negotiation.cc:407] Connection header received
0926 03:06:43.407025 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:43.407029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:43.407106 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:43.407217 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:43.408787 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:43.409633 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:43.411490 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:43.411879 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:43.411984 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:43.415463 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:43.415469 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:43.415473 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:43.415507 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:43.417981 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:43.418908 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:43.418917 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:43.418919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:43.419058 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:43.419481 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:43.419485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:43.419487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:43.419748 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:43.419855 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:43.420546 (+ 691us) server_negotiation.cc:299] Negotiation successful
0926 03:06:43.420714 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:06:43.421945 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:43.404131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.97.3:58665)
0926 03:06:43.405029 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:43.405078 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:43.406741 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:43.407392 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:43.407402 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:43.408028 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:43.408601 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:43.408618 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:43.409797 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:43.409801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:43.411335 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:43.411347 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:43.412089 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:43.412096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:43.412383 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:43.413008 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:43.413030 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:43.415234 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:43.418157 (+ 2923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:43.418165 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:43.418180 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:43.418749 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:43.419191 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:43.419195 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:43.419197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:43.419386 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:43.419862 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:43.419876 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:43.420196 (+ 320us) client_negotiation.cc:769] Sending connection context
0926 03:06:43.420419 (+ 223us) client_negotiation.cc:240] Negotiation successful
0926 03:06:43.420809 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":381,"threads_started":1}
I20260926 03:06:43.423404 7014 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36297
I20260926 03:06:43.423707 7014 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:43.424459 7014 heartbeater.cc:511] Master 127.0.97.62:36297 requested a full tablet report, sending...
I20260926 03:06:43.425828 6558 ts_manager.cc:194] Registered new tserver with Master: 4065ff7c51bd4a2499e8dd30a4e51c81 (127.0.97.3:44419)
I20260926 03:06:43.426654 6558 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58665
I20260926 03:06:43.431938 6747 heartbeater.cc:503] Master 127.0.97.62:36297 was elected leader, sending a full tablet report...
I20260926 03:06:43.436625 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260926 03:06:43.482028 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38703
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36297,127.0.97.61:38703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:43.805696 7019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:43.806002 7019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:43.806115 7019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:43.816046 7019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:06:43.816170 7019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:43.816231 7019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:43.816310 7019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:06:43.816365 7019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:06:43.826838 7019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36297,127.0.97.61:38703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
W20260926 03:06:43.827215 7019 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:36297,127.0.97.61:38703'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 03:06:43.828315 7019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:43.829833 7019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:43.841889 7024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:43.841940 7027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:43.841861 7025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:43.842756 7019 server_base.cc:1034] running on GCE node
I20260926 03:06:43.843518 7019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:43.844921 7019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:43.846143 7019 hybrid_clock.cc:648] HybridClock initialized: now 1790392003846066 us; error 50 us; skew 500 ppm
I20260926 03:06:43.846467 7019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:43 dist-test-slave-hmbw krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392003, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:43.849749 7019 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260926 03:06:43.851409 7019 webserver.cc:460] Webserver started at http://127.0.97.61:43655/ using document root <none> and password file <none>
I20260926 03:06:43.852093 7019 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:43.852180 7019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:43.852433 7019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:43.855568 7019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-1/data/instance:
uuid: "8c0a7c8022f64e94b309e7d5256076eb"
format_stamp: "Formatted at 2026-09-26 03:06:43 on dist-test-slave-hmbw"
server_key: "a8d876f445f31881ae4d792542f38294"
server_key_iv: "c8b7246de50a7aa12ab305ca5db340ab"
server_key_version: "encryptionkey@0"
I20260926 03:06:43.856326 7019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-1/wal/instance:
uuid: "8c0a7c8022f64e94b309e7d5256076eb"
format_stamp: "Formatted at 2026-09-26 03:06:43 on dist-test-slave-hmbw"
server_key: "a8d876f445f31881ae4d792542f38294"
server_key_iv: "c8b7246de50a7aa12ab305ca5db340ab"
server_key_version: "encryptionkey@0"
I20260926 03:06:43.861635 7019 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:06:43.865216 7034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:43.866726 7019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:06:43.866880 7019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-1/wal
uuid: "8c0a7c8022f64e94b309e7d5256076eb"
format_stamp: "Formatted at 2026-09-26 03:06:43 on dist-test-slave-hmbw"
server_key: "a8d876f445f31881ae4d792542f38294"
server_key_iv: "c8b7246de50a7aa12ab305ca5db340ab"
server_key_version: "encryptionkey@0"
I20260926 03:06:43.867066 7019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:43.887167 7019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:43.892023 7019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:43.892271 7019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:43.910198 7019 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:38703
I20260926 03:06:43.910198 7085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:38703 every 8 connection(s)
I20260926 03:06:43.911556 7019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-1/data/info.pb
I20260926 03:06:43.915388 7086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:43.920892 6881 heartbeater.cc:503] Master 127.0.97.62:36297 was elected leader, sending a full tablet report...
I20260926 03:06:43.921856 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 7019
I20260926 03:06:43.922362 7086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:06:43 dist-test-slave-hmbw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392003, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 26 03:06:43 dist-test-slave-hmbw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 03:06:43.943989 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:43.925639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44186 (local address 127.0.97.62:36297)
0926 03:06:43.926061 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:43.926067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:43.926677 (+ 610us) server_negotiation.cc:407] Connection header received
0926 03:06:43.927617 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:43.927622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:43.927694 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:43.927821 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:43.929567 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:43.930741 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:43.932886 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:43.933519 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:43.933654 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:43.937838 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:43.937860 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:43.937881 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:43.937925 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:43.940930 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:43.941782 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:43.941787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:43.941789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:43.941911 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:43.942296 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:43.942300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:43.942304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:43.942635 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:43.942762 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:43.943434 (+ 672us) server_negotiation.cc:299] Negotiation successful
0926 03:06:43.943648 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":210,"threads_started":1}
I20260926 03:06:43.945019 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:43.925620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.0.1:44186)
0926 03:06:43.926529 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:43.926581 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:43.927370 (+ 789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:43.927955 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:43.927961 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:43.928823 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:43.929341 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:43.929366 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:43.930951 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:43.930955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:43.932706 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:43.932721 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:43.933791 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:43.933797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:43.934028 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:43.934872 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:43.934911 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:43.937573 (+ 2662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:43.941053 (+ 3480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:43.941062 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:43.941073 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:43.941663 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:43.942014 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:43.942020 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:43.942024 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:43.942192 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:43.942748 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:43.942756 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:43.943091 (+ 335us) client_negotiation.cc:769] Sending connection context
0926 03:06:43.943349 (+ 258us) client_negotiation.cc:240] Negotiation successful
0926 03:06:43.943846 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":5,"spinlock_wait_cycles":16256,"thread_start_us":451,"threads_started":1}
I20260926 03:06:43.945036 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:43.924466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33710 (local address 127.0.97.61:38703)
0926 03:06:43.925274 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:43.925287 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:43.925327 (+ 40us) server_negotiation.cc:407] Connection header received
0926 03:06:43.927356 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:43.927379 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:43.927644 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:43.927873 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:43.929671 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:43.931283 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:43.932990 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:43.933837 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:43.934025 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:43.938341 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:43.938362 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:43.938380 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:43.938423 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:43.941649 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:43.942361 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:43.942375 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:43.942377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:43.942518 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:43.942869 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:43.942874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:43.942878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:43.943218 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:43.943470 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:43.943772 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 03:06:43.943998 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":626,"spinlock_wait_cycles":22912,"thread_start_us":401,"threads_started":1}
I20260926 03:06:43.946779 7086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:06:43 dist-test-slave-hmbw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392003, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 03:06:43.957908 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:43.947387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38703 (local address 127.0.0.1:33712)
0926 03:06:43.947553 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:43.947570 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:43.947646 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:43.947976 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:43.947980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:43.948283 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:43.948426 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:43.948434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:43.949337 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:43.949341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:43.950193 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:43.950203 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:43.950878 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:43.950882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:43.951037 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:43.951555 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:43.951571 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:43.953548 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:43.956027 (+ 2479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:43.956046 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:43.956049 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:43.956395 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:43.956683 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:43.956686 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:43.956688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:43.956810 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:43.957244 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:43.957247 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:43.957370 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 03:06:43.957473 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 03:06:43.957672 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 03:06:43.958076 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:43.947575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33712 (local address 127.0.97.61:38703)
0926 03:06:43.947723 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:43.947728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:43.947742 (+ 14us) server_negotiation.cc:407] Connection header received
0926 03:06:43.947783 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:43.947787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:43.947852 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:43.947928 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:43.948539 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:43.949235 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:43.950295 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:43.950733 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:43.950828 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:43.953707 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:43.953712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:43.953715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:43.953743 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:43.955916 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:43.956487 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:43.956491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:43.956493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:43.956597 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:43.956893 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:43.956897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:43.956900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:43.957158 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:43.957259 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:43.957601 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 03:06:43.957808 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 03:06:43.962865 7086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb: Bootstrap starting.
I20260926 03:06:43.965961 7086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:43.967007 7086 log.cc:824] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:43.969569 7086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb: No bootstrap required, opened a new log
I20260926 03:06:43.976724 7086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } }
I20260926 03:06:43.977068 7086 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:43.977145 7086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c0a7c8022f64e94b309e7d5256076eb, State: Initialized, Role: FOLLOWER
I20260926 03:06:43.977694 7086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } }
I20260926 03:06:43.978461 7093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } } }
I20260926 03:06:43.978777 7093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:06:43.979496 7086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [sys.catalog]: configured and running, proceeding with master startup.
W20260926 03:06:43.987846 7104 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:06:43.988080 7104 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:06:43.990439 7019 master_runner.cc:428] Detected that this master 8c0a7c8022f64e94b309e7d5256076eb is joining an existing cluster
I20260926 03:06:43.990562 7019 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:38703
I20260926 03:06:44.013164 7110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:43.994748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.0.1:44202)
0926 03:06:43.995408 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:43.995429 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:43.995586 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:43.996221 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:43.996226 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:43.996968 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:43.997381 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:43.997396 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:43.998870 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:43.998875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:44.000703 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:44.000718 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.001930 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.001937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:44.002110 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:44.002795 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:44.002820 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:44.004010 (+ 1190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:44.009796 (+ 5786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:44.009803 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:44.009807 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:44.010432 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:44.010948 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:44.010953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:44.010957 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:44.011127 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:44.011893 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:44.011898 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:44.012121 (+ 223us) client_negotiation.cc:769] Sending connection context
0926 03:06:44.012295 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 03:06:44.012647 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":27,"thread_start_us":370,"threads_started":1}
I20260926 03:06:44.014351 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:43.994742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44202 (local address 127.0.97.62:36297)
0926 03:06:43.994939 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:43.994945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:43.995537 (+ 592us) server_negotiation.cc:407] Connection header received
0926 03:06:43.995737 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:43.995743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:43.995896 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:43.995979 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:43.997558 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:43.998686 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.000894 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.001548 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.001674 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:44.004245 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:44.004252 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:44.004257 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:44.004299 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:44.009584 (+ 5285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:44.010638 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:44.010644 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:44.010648 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:44.010809 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:44.011268 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:44.011274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:44.011277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:44.011684 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:44.011833 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:44.013776 (+ 1943us) server_negotiation.cc:299] Negotiation successful
0926 03:06:44.014032 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:06:44.013706 7111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:43.995687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38703 (local address 127.0.0.1:33722)
0926 03:06:43.997926 (+ 2239us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:43.997942 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:43.998274 (+ 332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:43.998727 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:43.998731 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:43.999425 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:43.999821 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:43.999839 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.001240 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.001245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:44.002824 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:44.002840 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.004375 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.004381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:44.004559 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:44.005161 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:44.005183 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:44.006158 (+ 975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:44.011167 (+ 5009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:44.011171 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:44.011176 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:44.011684 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:44.012083 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:44.012088 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:44.012091 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:44.012228 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:44.012790 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:44.012794 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:44.012921 (+ 127us) client_negotiation.cc:769] Sending connection context
0926 03:06:44.013020 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 03:06:44.013299 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2124,"mutex_wait_us":35,"thread_start_us":1963,"threads_started":1}
I20260926 03:06:44.014467 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:43.995679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33722 (local address 127.0.97.61:38703)
0926 03:06:43.995866 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:43.995875 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:43.998020 (+ 2145us) server_negotiation.cc:407] Connection header received
0926 03:06:43.998402 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:43.998406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:43.998474 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:43.998560 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:44.000007 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.001048 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.002994 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.003680 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.003770 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:44.006320 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:44.006327 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:44.006332 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:44.006379 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:44.011030 (+ 4651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:44.011829 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:44.011834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:44.011836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:44.011975 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:44.012338 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:44.012353 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:44.012356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:44.012685 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:44.013843 (+ 1158us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:44.013950 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 03:06:44.014145 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":56}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Sep 26 03:06:44 dist-test-slave-hmbw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392001, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 03:06:44.038611 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:44.022434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33732 (local address 127.0.97.61:38703)
0926 03:06:44.022682 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:44.022689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:44.023425 (+ 736us) server_negotiation.cc:407] Connection header received
0926 03:06:44.023864 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:44.023870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:44.024009 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:44.024156 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:44.025352 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.026456 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.028346 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.028831 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.028966 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:44.032475 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:44.032484 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:44.032487 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:44.032522 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:44.035585 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:44.036401 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:44.036405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:44.036407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:44.036547 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:44.036956 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:44.036960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:44.036963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:44.037265 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:44.037406 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:44.037899 (+ 493us) server_negotiation.cc:299] Negotiation successful
0926 03:06:44.038213 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260926 03:06:44.038723 7113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:44.022299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38703 (local address 127.0.0.1:33732)
0926 03:06:44.023247 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:44.023314 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:44.023602 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:44.024214 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:44.024221 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:44.024913 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:44.025170 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:44.025182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.026662 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.026668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:44.028091 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:44.028114 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.029007 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.029013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:44.029210 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:44.029918 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:44.029937 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:44.032258 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:44.035799 (+ 3541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:44.035812 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:44.035820 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:44.036240 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:44.036661 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:44.036667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:44.036671 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:44.036854 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:44.037406 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:44.037415 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:44.037607 (+ 192us) client_negotiation.cc:769] Sending connection context
0926 03:06:44.037791 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 03:06:44.038199 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":510,"threads_started":1}
I20260926 03:06:44.040760 6558 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:38703: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 38703
}
attrs {
promote: true
}
}
dest_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71"
cas_config_opid_index: -1
I20260926 03:06:44.041755 6558 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: true } }
I20260926 03:06:44.055285 7116 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8c0a7c8022f64e94b309e7d5256076eb (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: true } } }
I20260926 03:06:44.057127 7061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:44.059434 7116 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:38703
I20260926 03:06:44.060063 7117 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: true } } }
I20260926 03:06:44.060653 7019 master.cc:537] Master@127.0.97.61:38703 shutting down...
I20260926 03:06:44.061162 7117 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260926 03:06:44.067140 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:36961
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36297,127.0.97.61:38703,127.0.97.60:36961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:06:44.067849 7061 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [term 1 FOLLOWER]: Refusing update from remote peer d8e4b0a3ad2449c1be9c81c740b8ef71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 03:06:44.073450 7019 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:06:44.073596 7117 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:44.074049 7019 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:44.074307 7019 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb: stopping tablet replica
W20260926 03:06:44.076782 6542 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260926 03:06:44.081671 7019 master.cc:552] Master@127.0.97.61:38703 shutdown complete.
I20260926 03:06:44.081861 7019 master_runner.cc:305] Clearing existing system tablet
I20260926 03:06:44.082387 7019 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:44.082669 7019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260926 03:06:44.088325 6542 consensus_peers.cc:487] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 -> Peer 8c0a7c8022f64e94b309e7d5256076eb (127.0.97.61:38703): Couldn't send request to peer 8c0a7c8022f64e94b309e7d5256076eb. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260926 03:06:44.089282 7019 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.004s
I20260926 03:06:44.091109 7122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:44.091439 7019 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:44.091961 7019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 03:06:44.092094 7019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-1/wal
uuid: "8c0a7c8022f64e94b309e7d5256076eb"
format_stamp: "Formatted at 2026-09-26 03:06:43 on dist-test-slave-hmbw"
server_key: "a8d876f445f31881ae4d792542f38294"
server_key_iv: "c8b7246de50a7aa12ab305ca5db340ab"
server_key_version: "encryptionkey@0"
I20260926 03:06:44.097271 7019 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:44.104673 7019 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 03:06:44.104995 7019 log.cc:1196] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 03:06:44.105537 7019 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb: Deleting consensus metadata
I20260926 03:06:44.105850 7019 master_runner.cc:315] Copying system tablet from 127.0.97.62:36297
I20260926 03:06:44.108351 7019 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:36297
I20260926 03:06:44.124253 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:44.109528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44204 (local address 127.0.97.62:36297)
0926 03:06:44.109708 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:44.109714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:44.110113 (+ 399us) server_negotiation.cc:407] Connection header received
0926 03:06:44.110291 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:44.110297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:44.110390 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:44.110529 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:44.111567 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.112681 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.114409 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.115105 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.115233 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:44.117536 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:44.117543 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:44.117548 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:44.117589 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:44.121185 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:44.122020 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:44.122027 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:44.122031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:44.122196 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:44.122642 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:44.122647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:44.122651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:44.123032 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:44.123194 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:44.123613 (+ 419us) server_negotiation.cc:299] Negotiation successful
0926 03:06:44.123856 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:06:44.124346 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:44.109129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.0.1:44204)
0926 03:06:44.109992 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:44.110011 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:44.110169 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:44.110562 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:44.110566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:44.111109 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:44.111416 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:44.111428 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.112832 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.112839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:44.114257 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:44.114272 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.115273 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.115280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:44.115463 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:44.116170 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:44.116192 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:44.117296 (+ 1104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:44.121363 (+ 4067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:44.121368 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:44.121373 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:44.121880 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:44.122321 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:44.122326 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:44.122330 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:44.122526 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:44.123162 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:44.123171 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:44.123339 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 03:06:44.123476 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 03:06:44.123799 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"thread_start_us":651,"threads_started":1}
I20260926 03:06:44.125605 6578 tablet_copy_service.cc:140] P d8e4b0a3ad2449c1be9c81c740b8ef71: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8c0a7c8022f64e94b309e7d5256076eb ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:44204)
I20260926 03:06:44.125878 6578 tablet_copy_service.cc:161] P d8e4b0a3ad2449c1be9c81c740b8ef71: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8c0a7c8022f64e94b309e7d5256076eb at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:44204: session id = 8c0a7c8022f64e94b309e7d5256076eb-00000000000000000000000000000000
I20260926 03:06:44.130182 6578 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 03:06:44.132567 7019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 03:06:44.139565 7019 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb: tablet copy: Starting download of 0 data blocks...
I20260926 03:06:44.139902 7019 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb: tablet copy: Starting download of 1 WAL segments...
I20260926 03:06:44.143014 7019 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 03:06:44.147549 6578 tablet_copy_service.cc:342] P d8e4b0a3ad2449c1be9c81c740b8ef71: Request end of tablet copy session 8c0a7c8022f64e94b309e7d5256076eb-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:44204
I20260926 03:06:44.147785 6578 tablet_copy_service.cc:433] P d8e4b0a3ad2449c1be9c81c740b8ef71: ending tablet copy session 8c0a7c8022f64e94b309e7d5256076eb-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8c0a7c8022f64e94b309e7d5256076eb
I20260926 03:06:44.159751 7019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:44.162581 7129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:44.163256 7130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:44.170001 7132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:44.171849 7019 server_base.cc:1034] running on GCE node
I20260926 03:06:44.172225 7019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:44.172884 7019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:44.174180 7019 hybrid_clock.cc:648] HybridClock initialized: now 1790392004174109 us; error 36 us; skew 500 ppm
I20260926 03:06:44.174404 7019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:44 dist-test-slave-hmbw krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392004, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:44.176652 7019 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260926 03:06:44.177927 7019 webserver.cc:460] Webserver started at http://127.0.97.61:43999/ using document root <none> and password file <none>
I20260926 03:06:44.178284 7019 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:44.178370 7019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:44.181489 7019 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 03:06:44.183485 7139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:44.183928 7019 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260926 03:06:44.184109 7019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-1/wal
uuid: "8c0a7c8022f64e94b309e7d5256076eb"
format_stamp: "Formatted at 2026-09-26 03:06:43 on dist-test-slave-hmbw"
server_key: "a8d876f445f31881ae4d792542f38294"
server_key_iv: "c8b7246de50a7aa12ab305ca5db340ab"
server_key_version: "encryptionkey@0"
I20260926 03:06:44.184331 7019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:44.211992 7019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:44.216154 7019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:44.237569 7019 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:38703
I20260926 03:06:44.237612 7190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:38703 every 8 connection(s)
I20260926 03:06:44.238694 7019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-1/data/info.pb
I20260926 03:06:44.244026 7191 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:06:44.248087 7191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb: Bootstrap starting.
I20260926 03:06:44.265285 7191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:06:44.265834 7191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb: Bootstrap complete.
I20260926 03:06:44.266866 7191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: true } }
I20260926 03:06:44.267123 7191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8c0a7c8022f64e94b309e7d5256076eb, State: Initialized, Role: LEARNER
I20260926 03:06:44.267422 7191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: true } }
I20260926 03:06:44.268239 7194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: true } } }
I20260926 03:06:44.268493 7191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:06:44.268630 7194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [sys.catalog]: This master's current role is: LEARNER
I20260926 03:06:44.279645 7205 catalog_manager.cc:1259] Loaded cluster ID: 6aeb71c0cc784ff4b8ab8bd34d9dd50f
I20260926 03:06:44.279752 7205 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb: loading cluster ID for follower catalog manager: success
I20260926 03:06:44.283291 7205 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb: acquiring CA information for follower catalog manager: success
I20260926 03:06:44.285861 7205 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:06:44.429134 7014 heartbeater.cc:503] Master 127.0.97.62:36297 was elected leader, sending a full tablet report...
W20260926 03:06:44.434538 7119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:44.434867 7119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:44.435006 7119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:44.442713 7119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:06:44.442816 7119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:44.442864 7119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:44.442916 7119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:06:44.442956 7119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:06:44.453477 7119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36297,127.0.97.61:38703,127.0.97.60:36961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:36961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:44.454852 7119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:44.456128 7119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:44.467402 7211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:44.467809 7213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:44.468632 7210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:44.468835 7119 server_base.cc:1034] running on GCE node
I20260926 03:06:44.469642 7119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:44.471127 7119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:44.472317 7119 hybrid_clock.cc:648] HybridClock initialized: now 1790392004472272 us; error 39 us; skew 500 ppm
I20260926 03:06:44.472641 7119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:44 dist-test-slave-hmbw krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392004, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:44.481222 7119 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260926 03:06:44.482695 7119 webserver.cc:460] Webserver started at http://127.0.97.60:36661/ using document root <none> and password file <none>
I20260926 03:06:44.483369 7119 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:44.483484 7119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:44.483762 7119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:44.486819 7119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-2/data/instance:
uuid: "04ff140057b04287990399ccb2cc2436"
format_stamp: "Formatted at 2026-09-26 03:06:44 on dist-test-slave-hmbw"
server_key: "48cf3c1122fb7520a5ce9ed87ecfad67"
server_key_iv: "589469e9bb953bf87c5536916129f4d7"
server_key_version: "encryptionkey@0"
I20260926 03:06:44.487589 7119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-2/wal/instance:
uuid: "04ff140057b04287990399ccb2cc2436"
format_stamp: "Formatted at 2026-09-26 03:06:44 on dist-test-slave-hmbw"
server_key: "48cf3c1122fb7520a5ce9ed87ecfad67"
server_key_iv: "589469e9bb953bf87c5536916129f4d7"
server_key_version: "encryptionkey@0"
I20260926 03:06:44.492651 7119 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 03:06:44.496186 7220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:44.497694 7119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:06:44.497888 7119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-2/wal
uuid: "04ff140057b04287990399ccb2cc2436"
format_stamp: "Formatted at 2026-09-26 03:06:44 on dist-test-slave-hmbw"
server_key: "48cf3c1122fb7520a5ce9ed87ecfad67"
server_key_iv: "589469e9bb953bf87c5536916129f4d7"
server_key_version: "encryptionkey@0"
I20260926 03:06:44.498078 7119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:44.519541 7119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:44.523662 7119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:44.523933 7119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:44.543547 7119 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:36961
I20260926 03:06:44.543565 7272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:36961 every 8 connection(s)
I20260926 03:06:44.544400 7113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:44.537421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38703 (local address 127.0.0.1:39342)
0926 03:06:44.537622 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:44.537639 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:44.537748 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:44.538409 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:44.538415 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:44.538451 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:44.538683 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:44.538695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.540240 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.540247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:44.542225 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:44.542239 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.543608 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.543613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:44.543846 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:44.543901 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 03:06:44.543990 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 03:06:44.544127 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 03:06:44.544449 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:44.537558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39342 (local address 127.0.97.61:38703)
0926 03:06:44.538044 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:44.538049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:44.538070 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:06:44.538136 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:44.538141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:44.538215 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:44.538334 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:44.538833 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.540098 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.542402 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.543448 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.543569 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:44.543787 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:44.544028 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 03:06:44.544156 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":287,"threads_started":1}
I20260926 03:06:44.545014 7119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-2/data/info.pb
I20260926 03:06:44.545634 7166 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260926 03:06:44.546198 7194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8e4b0a3ad2449c1be9c81c740b8ef71. Latest consensus state: current_term: 1 leader_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: true } } }
I20260926 03:06:44.546422 7194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [sys.catalog]: This master's current role is: LEARNER
I20260926 03:06:44.548889 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 7119
I20260926 03:06:44.550140 7273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:44.559726 7273 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:06:44 dist-test-slave-hmbw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 26 03:06:44 dist-test-slave-hmbw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392004, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:06:44.572986 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:44.551225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56906 (local address 127.0.97.60:36961)
0926 03:06:44.552007 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:44.552021 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:44.552068 (+ 47us) server_negotiation.cc:407] Connection header received
0926 03:06:44.553714 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:44.553741 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:44.554272 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:44.554732 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:44.556105 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.558086 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.559400 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.560415 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.560606 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:44.565590 (+ 4984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:44.565601 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:44.565614 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:44.565664 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:44.568750 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:44.569539 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:44.569545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:44.569552 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:44.569719 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:44.570142 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:44.570145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:44.570147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:44.570742 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:44.571016 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:44.571392 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 03:06:44.571836 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":314,"threads_started":1}
I20260926 03:06:44.577436 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:44.562880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55972 (local address 127.0.97.62:36297)
0926 03:06:44.563048 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:44.563053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:44.564083 (+ 1030us) server_negotiation.cc:407] Connection header received
0926 03:06:44.564268 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:44.564272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:44.564335 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:44.564416 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:44.565894 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.566901 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.568350 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.568766 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.568885 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:44.572327 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:44.572332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:44.572335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:44.572367 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:44.574973 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:44.575753 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:44.575758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:44.575761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:44.575892 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:44.576215 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:44.576221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:44.576223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:44.576531 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:44.576654 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:44.577038 (+ 384us) server_negotiation.cc:299] Negotiation successful
0926 03:06:44.577201 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 03:06:44.577467 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:44.562765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.0.1:55972)
0926 03:06:44.563924 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:44.563976 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:44.564149 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:44.564523 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:44.564531 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:44.565344 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:44.565743 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:44.565759 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.567042 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.567048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:44.568217 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:44.568232 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.568941 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.568946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:44.569071 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:44.569619 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:44.569642 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:44.572121 (+ 2479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:44.575094 (+ 2973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:44.575102 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:44.575108 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:44.575635 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:44.575979 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:44.575982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:44.575984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:44.576091 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:44.576628 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:44.576634 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:44.576782 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 03:06:44.576965 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 03:06:44.577197 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"thread_start_us":783,"threads_started":1}
I20260926 03:06:44.579258 7273 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:06:44 dist-test-slave-hmbw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392004, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 03:06:44.590206 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:44.579791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38703 (local address 127.0.0.1:39350)
0926 03:06:44.579971 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:44.579989 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:44.580081 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:44.580381 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:44.580384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:44.580664 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:44.580790 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:44.580797 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.581680 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.581683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:44.582638 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:44.582649 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.583235 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.583239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:44.583365 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:44.583825 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:44.583840 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:44.585731 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:44.588323 (+ 2592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:44.588327 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:44.588330 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:44.588680 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:44.588985 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:44.588989 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:44.588991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:44.589108 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:44.589548 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:44.589552 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:44.589674 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 03:06:44.589778 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 03:06:44.589996 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 03:06:44.590252 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:44.579998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39350 (local address 127.0.97.61:38703)
0926 03:06:44.580154 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:44.580158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:44.580172 (+ 14us) server_negotiation.cc:407] Connection header received
0926 03:06:44.580219 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:44.580222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:44.580277 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:44.580348 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:44.580900 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.581583 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.582737 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.583129 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.583214 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:44.585890 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:44.585896 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:44.585901 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:44.585936 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:44.588202 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:44.588765 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:44.588769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:44.588771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:44.588904 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:44.589192 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:44.589196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:44.589198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:44.589448 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:44.589556 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:44.589850 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 03:06:44.590034 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 03:06:44.591197 7273 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36961 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:06:44 dist-test-slave-hmbw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392004, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260926 03:06:44.602478 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:44.591801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36961 (local address 127.0.0.1:56922)
0926 03:06:44.591993 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:44.592009 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:44.592102 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:44.592344 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:44.592348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:44.592681 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:44.592808 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:44.592816 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.593694 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.593699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:44.594600 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:44.594612 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.595220 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.595226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:44.595396 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:44.595894 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:44.595911 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:44.597678 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:44.600101 (+ 2423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:44.600106 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:44.600110 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:44.600830 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:44.601219 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:44.601222 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:44.601224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:44.601357 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:44.601763 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:44.601766 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:44.601910 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 03:06:44.602011 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 03:06:44.602210 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 03:06:44.602756 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:44.591810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56922 (local address 127.0.97.60:36961)
0926 03:06:44.591961 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:44.591970 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:44.592070 (+ 100us) server_negotiation.cc:407] Connection header received
0926 03:06:44.592200 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:44.592204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:44.592250 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:44.592319 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:44.592931 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.593588 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.594713 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.595089 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.595185 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:44.597825 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:44.597830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:44.597833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:44.597861 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:44.599991 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:44.600992 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:44.600998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:44.601002 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:44.601122 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:44.601438 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:44.601442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:44.601444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:44.601678 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:44.601777 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:44.602100 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 03:06:44.602344 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":43}
I20260926 03:06:44.607296 7273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436: Bootstrap starting.
I20260926 03:06:44.610363 7273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:44.611364 7273 log.cc:824] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:44.613797 7273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436: No bootstrap required, opened a new log
I20260926 03:06:44.621152 7273 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36961 } }
I20260926 03:06:44.621503 7273 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:44.621608 7273 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04ff140057b04287990399ccb2cc2436, State: Initialized, Role: FOLLOWER
I20260926 03:06:44.622184 7273 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36961 } }
I20260926 03:06:44.623063 7279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36961 } } }
I20260926 03:06:44.623381 7279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:06:44.624114 7273 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 03:06:44.633671 7290 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:06:44.633839 7290 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:06:44.638051 7119 master_runner.cc:428] Detected that this master 04ff140057b04287990399ccb2cc2436 is joining an existing cluster
I20260926 03:06:44.638183 7119 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:36961
I20260926 03:06:44.656710 7279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:06:44.657016 7279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36961 } }
I20260926 03:06:44.658594 7279 leader_election.cc:288] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8e4b0a3ad2449c1be9c81c740b8ef71 (127.0.97.62:36297), 8c0a7c8022f64e94b309e7d5256076eb (127.0.97.61:38703)
I20260926 03:06:44.659761 6568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04ff140057b04287990399ccb2cc2436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" is_pre_election: true
I20260926 03:06:44.660619 7166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04ff140057b04287990399ccb2cc2436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c0a7c8022f64e94b309e7d5256076eb" is_pre_election: true
I20260926 03:06:44.663720 7221 leader_election.cc:302] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 04ff140057b04287990399ccb2cc2436; no voters: 8c0a7c8022f64e94b309e7d5256076eb, d8e4b0a3ad2449c1be9c81c740b8ef71
I20260926 03:06:44.665686 7279 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 03:06:44.670428 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:44.643378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55978 (local address 127.0.97.62:36297)
0926 03:06:44.643558 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:44.643564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:44.643999 (+ 435us) server_negotiation.cc:407] Connection header received
0926 03:06:44.644258 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:44.644264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:44.644340 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:44.644422 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:44.646235 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.647506 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.649880 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.650630 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.650733 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:44.653817 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:44.653825 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:44.653830 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:44.653877 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:44.667066 (+ 13189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:44.667944 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:44.667951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:44.667955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:44.668122 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:44.668675 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:44.668681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:44.668685 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:44.669037 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:44.669204 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:44.669844 (+ 640us) server_negotiation.cc:299] Negotiation successful
0926 03:06:44.670097 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:06:44.671526 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:44.643036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56934 (local address 127.0.97.60:36961)
0926 03:06:44.643485 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:44.643491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:44.655677 (+ 12186us) server_negotiation.cc:407] Connection header received
0926 03:06:44.656134 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:44.656140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:44.656211 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:44.656275 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:44.657746 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.658803 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.660690 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.661320 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.661410 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:44.664494 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:44.664500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:44.664504 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:44.664540 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:44.668190 (+ 3650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:44.668906 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:44.668911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:44.668915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:44.669051 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:44.669417 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:44.669421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:44.669424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:44.669756 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:44.670373 (+ 617us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:44.670935 (+ 562us) server_negotiation.cc:299] Negotiation successful
0926 03:06:44.671170 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310}
I20260926 03:06:44.672245 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:44.642937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.0.1:55978)
0926 03:06:44.643898 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:44.643916 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:44.644104 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:44.644664 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:44.644669 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:44.645470 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:44.646031 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:44.646047 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.647726 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.647732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:44.649676 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:44.649700 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.651113 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.651120 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:44.651318 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:44.652257 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:44.652284 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:44.653549 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:44.667266 (+ 13717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:44.667275 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:44.667279 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:44.667791 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:44.668262 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:44.668266 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:44.668270 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:44.668480 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:44.669293 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:44.669297 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:44.669527 (+ 230us) client_negotiation.cc:769] Sending connection context
0926 03:06:44.670995 (+ 1468us) client_negotiation.cc:240] Negotiation successful
0926 03:06:44.671344 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":32,"thread_start_us":685,"threads_started":1}
I20260926 03:06:44.672478 7297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:44.643076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36961 (local address 127.0.0.1:56934)
0926 03:06:44.655579 (+ 12503us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:44.655600 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:44.655999 (+ 399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:44.656465 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:44.656470 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:44.657234 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:44.657594 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:44.657607 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.658996 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.659001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:44.660544 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:44.660557 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.661735 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.661743 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:44.661966 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:44.663084 (+ 1118us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:44.663106 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:44.664299 (+ 1193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:44.668312 (+ 4013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:44.668316 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:44.668319 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:44.668786 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:44.669166 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:44.669170 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:44.669173 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:44.669320 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:44.670470 (+ 1150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:44.670474 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:44.670677 (+ 203us) client_negotiation.cc:769] Sending connection context
0926 03:06:44.671708 (+ 1031us) client_negotiation.cc:240] Negotiation successful
0926 03:06:44.672018 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12325,"mutex_wait_us":1,"thread_start_us":12604,"threads_started":1}
I20260926 03:06:44.674603 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:44.644394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39360 (local address 127.0.97.61:38703)
0926 03:06:44.644570 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:44.644576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:44.656913 (+ 12337us) server_negotiation.cc:407] Connection header received
0926 03:06:44.658806 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:44.658812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:44.658906 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:44.658976 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:44.660450 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.661559 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.664206 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.664857 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.664967 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:44.667558 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:44.667566 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:44.667571 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:44.667613 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:44.671184 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:44.672043 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:44.672049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:44.672052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:44.672208 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:44.672662 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:44.672668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:44.672671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:44.673028 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:44.673167 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:44.673979 (+ 812us) server_negotiation.cc:299] Negotiation successful
0926 03:06:44.674229 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:06:44.675287 7298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:44.644078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38703 (local address 127.0.0.1:39360)
0926 03:06:44.656794 (+ 12716us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:44.656817 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:44.658630 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:44.659176 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:44.659181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:44.659887 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:44.660264 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:44.660276 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.661747 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.661753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:44.664014 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:44.664030 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.665265 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.665272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:44.665464 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:44.666106 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:44.666130 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:44.667318 (+ 1188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:44.671372 (+ 4054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:44.671377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:44.671384 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:44.671883 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:44.672350 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:44.672355 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:44.672359 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:44.672535 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:44.673422 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:44.673427 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:44.673638 (+ 211us) client_negotiation.cc:769] Sending connection context
0926 03:06:44.673820 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 03:06:44.674223 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12563,"mutex_wait_us":27,"thread_start_us":12412,"threads_started":1}
I20260926 03:06:44.678471 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 6619
Sep 26 03:06:44 dist-test-slave-hmbw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392001, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 26 03:06:44 dist-test-slave-hmbw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260926 03:06:44.695448 7113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:44.678958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36961 (local address 127.0.0.1:56946)
0926 03:06:44.679137 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:44.679156 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:44.679260 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:44.679688 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:44.679693 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:44.680167 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:44.680352 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:44.680364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.681914 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.681919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:44.683466 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:44.683480 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.684255 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.684262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:44.684437 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:44.685138 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:44.685160 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:44.687868 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:44.691946 (+ 4078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:44.691953 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:44.691957 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:44.692359 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:44.693545 (+ 1186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:44.693550 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:44.693554 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:44.693733 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:44.694400 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:44.694405 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:44.694596 (+ 191us) client_negotiation.cc:769] Sending connection context
0926 03:06:44.694758 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 03:06:44.695057 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 03:06:44.695451 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:44.679157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56946 (local address 127.0.97.60:36961)
0926 03:06:44.679323 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:44.679328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:44.679347 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:06:44.679437 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:44.679443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:44.679523 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:44.680494 (+ 971us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:44.680676 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.681759 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.683600 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.684055 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.684163 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:44.688106 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:44.688113 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:44.688118 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:44.688156 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:44.691815 (+ 3659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:44.693259 (+ 1444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:44.693265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:44.693268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:44.693407 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:44.693849 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:44.693854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:44.693857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:44.694261 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:44.694415 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:44.694869 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 03:06:44.695104 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:06:44.697350 6558 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:36961: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "04ff140057b04287990399ccb2cc2436"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 36961
}
attrs {
promote: true
}
}
dest_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71"
cas_config_opid_index: 5
I20260926 03:06:44.698397 7299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:44.679981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37090 (local address 127.0.97.1:43877)
0926 03:06:44.680543 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:44.680550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:44.680573 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:06:44.680622 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:44.680628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:44.680821 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:44.680974 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:44.682243 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.684187 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.685925 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.686903 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.687041 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:44.691437 (+ 4396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:44.691454 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:44.691473 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:44.691522 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:44.694959 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:44.695782 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:44.695790 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:44.695794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:44.695927 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:44.696692 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:44.696696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:44.696699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:44.697108 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:44.697396 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:44.697759 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 03:06:44.698037 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":270,"threads_started":1}
I20260926 03:06:44.698359 6558 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: true } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: true } }
I20260926 03:06:44.699626 6742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:44.702410 7247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:44.703325 7166 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [term 1 LEARNER]: Refusing update from remote peer d8e4b0a3ad2449c1be9c81c740b8ef71: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 03:06:44.704075 7117 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 03:06:44.710047 7247 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [term 1 FOLLOWER]: Refusing update from remote peer d8e4b0a3ad2449c1be9c81c740b8ef71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 03:06:44.712280 7302 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:44.700323 7300 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 04ff140057b04287990399ccb2cc2436 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: true } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: true } } }
I20260926 03:06:44.719384 7166 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 04ff140057b04287990399ccb2cc2436 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: true } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: true } } }
I20260926 03:06:44.725358 7194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: true } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: true } } }
I20260926 03:06:44.725659 7194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [sys.catalog]: This master's current role is: LEARNER
I20260926 03:06:44.726079 7247 raft_consensus.cc:858] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36961 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: true } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: true } }
W20260926 03:06:44.726413 7247 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36961 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: true } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: true } }
I20260926 03:06:44.728843 7300 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:36961
I20260926 03:06:44.729558 7119 master.cc:537] Master@127.0.97.60:36961 shutting down...
I20260926 03:06:44.731007 7117 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: true } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: true } } }
I20260926 03:06:44.731276 7117 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:44.737857 7302 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71: attempting to promote NON_VOTER 8c0a7c8022f64e94b309e7d5256076eb to VOTER
I20260926 03:06:44.738835 7302 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: true } }
I20260926 03:06:44.747220 7117 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: true } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: true } } }
I20260926 03:06:44.747711 7117 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:44.750586 7166 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [term 1 LEARNER]: Refusing update from remote peer d8e4b0a3ad2449c1be9c81c740b8ef71: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 03:06:44.751905 7307 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260926 03:06:44.753931 6543 consensus_peers.cc:487] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 -> Peer 04ff140057b04287990399ccb2cc2436 (127.0.97.60:36961): Couldn't send request to peer 04ff140057b04287990399ccb2cc2436. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 03:06:44.755352 7300 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8c0a7c8022f64e94b309e7d5256076eb (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: true } } }
I20260926 03:06:44.755453 7165 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8c0a7c8022f64e94b309e7d5256076eb (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: true } } }
I20260926 03:06:44.759918 7117 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: true } } }
I20260926 03:06:44.759953 7194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: true } } }
I20260926 03:06:44.760162 7117 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:44.760191 7194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:06:44.760423 7117 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: true } } }
I20260926 03:06:44.760602 7117 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:44.764357 7247 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8c0a7c8022f64e94b309e7d5256076eb (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: true } } }
I20260926 03:06:44.776489 7279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8e4b0a3ad2449c1be9c81c740b8ef71. Latest consensus state: current_term: 1 leader_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: true } } }
I20260926 03:06:44.776749 7279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 03:06:44.777102 7310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: true } } }
I20260926 03:06:44.777444 7310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 03:06:44.775667 7314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: true } } }
I20260926 03:06:44.777557 7314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 03:06:44.778330 7119 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260926 03:06:44.778792 7119 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260926 03:06:44.778961 7119 tablet_replica.cc:330] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436: stopping tablet replica
I20260926 03:06:44.786756 7119 master.cc:552] Master@127.0.97.60:36961 shutdown complete.
I20260926 03:06:44.786845 7119 master_runner.cc:305] Clearing existing system tablet
I20260926 03:06:44.787295 7119 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:44.787447 7119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:44.790361 7119 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260926 03:06:44.810871 7316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:44.814543 7119 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:44.814849 7119 fs_manager.cc:725] Time spent opening block manager: real 0.024s user 0.005s sys 0.015s
I20260926 03:06:44.815023 7119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-2/wal
uuid: "04ff140057b04287990399ccb2cc2436"
format_stamp: "Formatted at 2026-09-26 03:06:44 on dist-test-slave-hmbw"
server_key: "48cf3c1122fb7520a5ce9ed87ecfad67"
server_key_iv: "589469e9bb953bf87c5536916129f4d7"
server_key_version: "encryptionkey@0"
I20260926 03:06:44.819847 7119 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:44.827414 7119 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 03:06:44.827605 7119 log.cc:1196] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 03:06:44.828167 7119 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436: Deleting consensus metadata
I20260926 03:06:44.828433 7119 master_runner.cc:315] Copying system tablet from 127.0.97.62:36297
I20260926 03:06:44.831112 7119 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:36297
I20260926 03:06:44.849493 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 6619
I20260926 03:06:44.851174 7322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:44.831912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.0.1:55992)
0926 03:06:44.834967 (+ 3055us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:44.834991 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:44.835169 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:44.835571 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:44.835577 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:44.836173 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:44.836523 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:44.836538 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.838084 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.838090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:44.839675 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:44.839693 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.841140 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.841146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:44.841336 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:44.842030 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:44.842054 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:44.843283 (+ 1229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:44.847651 (+ 4368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:44.847782 (+ 131us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:44.847805 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:44.848234 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:44.848669 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:44.848674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:44.848677 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:44.848822 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:44.850044 (+ 1222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:44.850050 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:44.850241 (+ 191us) client_negotiation.cc:769] Sending connection context
0926 03:06:44.850379 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 03:06:44.850721 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2910,"thread_start_us":2756,"threads_started":1}
I20260926 03:06:44.852094 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:44.831952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55992 (local address 127.0.97.62:36297)
0926 03:06:44.832166 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:44.832173 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:44.835120 (+ 2947us) server_negotiation.cc:407] Connection header received
0926 03:06:44.835316 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:44.835323 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:44.835410 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:44.835522 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:44.836699 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.837860 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.839842 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:44.840932 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:44.841071 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:44.843515 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:44.843523 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:44.843528 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:44.843568 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:44.847357 (+ 3789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:44.848382 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:44.848388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:44.848392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:44.848556 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:44.849460 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:44.849465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:44.849469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:44.849832 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:44.849980 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:44.851510 (+ 1530us) server_negotiation.cc:299] Negotiation successful
0926 03:06:44.851758 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260926 03:06:44.852593 6578 tablet_copy_service.cc:140] P d8e4b0a3ad2449c1be9c81c740b8ef71: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 04ff140057b04287990399ccb2cc2436 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:55992)
I20260926 03:06:44.852772 6578 tablet_copy_service.cc:161] P d8e4b0a3ad2449c1be9c81c740b8ef71: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 04ff140057b04287990399ccb2cc2436 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:55992: session id = 04ff140057b04287990399ccb2cc2436-00000000000000000000000000000000
I20260926 03:06:44.856417 6578 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 03:06:44.858429 7119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 03:06:44.865361 7119 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436: tablet copy: Starting download of 0 data blocks...
I20260926 03:06:44.865698 7119 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436: tablet copy: Starting download of 1 WAL segments...
I20260926 03:06:44.868304 7119 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 03:06:44.868628 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43877
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:36297
--webserver_port=45957
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:36297,127.0.97.61:38703
--tserver_master_addrs=127.0.97.62:36297,127.0.97.61:38703,127.0.97.60:36961 with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:06:44.872920 6578 tablet_copy_service.cc:342] P d8e4b0a3ad2449c1be9c81c740b8ef71: Request end of tablet copy session 04ff140057b04287990399ccb2cc2436-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:55992
I20260926 03:06:44.873080 6578 tablet_copy_service.cc:433] P d8e4b0a3ad2449c1be9c81c740b8ef71: ending tablet copy session 04ff140057b04287990399ccb2cc2436-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 04ff140057b04287990399ccb2cc2436
I20260926 03:06:44.883004 7119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:44.885636 7324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:44.891755 7325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:44.896962 7327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:44.900096 7119 server_base.cc:1034] running on GCE node
I20260926 03:06:44.900439 7119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:44.901049 7119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:44.902222 7119 hybrid_clock.cc:648] HybridClock initialized: now 1790392004902167 us; error 52 us; skew 500 ppm
I20260926 03:06:44.902437 7119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:44 dist-test-slave-hmbw krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392004, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:44.904306 7119 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260926 03:06:44.911623 7119 webserver.cc:460] Webserver started at http://127.0.97.60:34637/ using document root <none> and password file <none>
I20260926 03:06:44.911911 7119 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:44.912003 7119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:44.914214 7119 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 03:06:44.915805 7334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:44.916170 7119 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 03:06:44.916282 7119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-2/wal
uuid: "04ff140057b04287990399ccb2cc2436"
format_stamp: "Formatted at 2026-09-26 03:06:44 on dist-test-slave-hmbw"
server_key: "48cf3c1122fb7520a5ce9ed87ecfad67"
server_key_iv: "589469e9bb953bf87c5536916129f4d7"
server_key_version: "encryptionkey@0"
I20260926 03:06:44.916451 7119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:44.931982 7119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:44.935308 7119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:44.974351 7119 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:36961
I20260926 03:06:44.974370 7385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:36961 every 8 connection(s)
I20260926 03:06:44.975306 7119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/master-2/data/info.pb
I20260926 03:06:44.981027 7386 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:06:44.985450 7386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436: Bootstrap starting.
I20260926 03:06:45.018862 7386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:06:45.019356 7386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436: Bootstrap complete.
I20260926 03:06:45.020256 7386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: true } }
I20260926 03:06:45.020478 7386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 04ff140057b04287990399ccb2cc2436, State: Initialized, Role: LEARNER
I20260926 03:06:45.020707 7386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: true } }
I20260926 03:06:45.021474 7389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: true } } }
I20260926 03:06:45.021766 7386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:06:45.021935 7389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [sys.catalog]: This master's current role is: LEARNER
I20260926 03:06:45.029026 7400 catalog_manager.cc:1259] Loaded cluster ID: 6aeb71c0cc784ff4b8ab8bd34d9dd50f
I20260926 03:06:45.029117 7400 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436: loading cluster ID for follower catalog manager: success
I20260926 03:06:45.031629 7400 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436: acquiring CA information for follower catalog manager: success
I20260926 03:06:45.033427 7400 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 03:06:45.300761 7323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:45.301186 7323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:45.301343 7323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 03:06:45.305927 7404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:45.299222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36961 (local address 127.0.0.1:56960)
0926 03:06:45.299717 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:45.299739 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:45.299877 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:45.300326 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:45.300329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:45.300350 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:45.300509 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:45.300517 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:45.302312 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:45.302317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:45.303649 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:45.303669 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:45.305323 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:45.305328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:45.305475 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:45.305503 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 03:06:45.305585 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 03:06:45.305691 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":234,"threads_started":1}
I20260926 03:06:45.306095 7405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:45.299334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56960 (local address 127.0.97.60:36961)
0926 03:06:45.299984 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:45.299990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:45.300012 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:06:45.300084 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:45.300089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:45.300185 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:45.300314 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:45.300650 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:45.302128 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:45.303863 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:45.305108 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:45.305262 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:45.305530 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:45.305618 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 03:06:45.305758 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":369,"threads_started":1}
I20260926 03:06:45.307006 7361 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 03:06:45.307456 7389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8e4b0a3ad2449c1be9c81c740b8ef71. Latest consensus state: current_term: 1 leader_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: true } } }
I20260926 03:06:45.307720 7389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [sys.catalog]: This master's current role is: LEARNER
W20260926 03:06:45.310644 7323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:45.310756 7323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:45.310878 7323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:06:45.321777 7323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45957
--tserver_master_addrs=127.0.97.62:36297,127.0.97.61:38703,127.0.97.60:36961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:45.323501 7323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:45.325006 7323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:45.338930 7407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:45.338930 7408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:45.339172 7410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:45.340165 7323 server_base.cc:1034] running on GCE node
I20260926 03:06:45.340875 7323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:45.341918 7323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:45.343143 7323 hybrid_clock.cc:648] HybridClock initialized: now 1790392005343077 us; error 56 us; skew 500 ppm
I20260926 03:06:45.343516 7323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:45 dist-test-slave-hmbw krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392005, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:45.346891 7323 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260926 03:06:45.348470 7323 webserver.cc:460] Webserver started at http://127.0.97.1:45957/ using document root <none> and password file <none>
I20260926 03:06:45.349175 7323 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:45.349309 7323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:45.355253 7323 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 03:06:45.358815 7417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:45.360476 7323 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:06:45.360661 7323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "1f35873792864b84a8ee278ba9b71a9d"
format_stamp: "Formatted at 2026-09-26 03:06:42 on dist-test-slave-hmbw"
server_key: "accc9a5378001f28e4d5d0808c814358"
server_key_iv: "4e9217140e5691d56762fb775cb690df"
server_key_version: "encryptionkey@0"
I20260926 03:06:45.361716 7323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:45.397389 7323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:45.402477 7323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:45.402766 7323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:45.403760 7323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:06:45.405200 7323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:45.405294 7323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:45.405404 7323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:45.405472 7323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:45.439630 7323 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43877
I20260926 03:06:45.439685 7529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43877 every 8 connection(s)
I20260926 03:06:45.441184 7323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:06:45.445742 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 7323
I20260926 03:06:45.446058 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 6753
Sep 26 03:06:45 dist-test-slave-hmbw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392005, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 26 03:06:45 dist-test-slave-hmbw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392005, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 26 03:06:45 dist-test-slave-hmbw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392005, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:06:45.471731 7534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:45.444484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44891 (local address 127.0.97.61:38703)
0926 03:06:45.445127 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:45.445134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:45.446013 (+ 879us) server_negotiation.cc:407] Connection header received
0926 03:06:45.449409 (+ 3396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:45.449415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:45.449537 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:45.449623 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:45.452771 (+ 3148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:45.454015 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:45.456851 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:45.457460 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:45.457552 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:45.464024 (+ 6472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:45.464033 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:45.464038 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:45.464078 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:45.467610 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:45.468651 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:45.468657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:45.468660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:45.468822 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:45.469465 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:45.469470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:45.469473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:45.469828 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:45.469977 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:45.471099 (+ 1122us) server_negotiation.cc:299] Negotiation successful
0926 03:06:45.471341 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":416,"threads_started":1}
I20260926 03:06:45.473274 7536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:45.444779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38703 (local address 127.0.97.1:44891)
0926 03:06:45.445826 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:45.445884 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:45.449005 (+ 3121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:45.450463 (+ 1458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:45.450477 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:45.451559 (+ 1082us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:45.452478 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:45.452508 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:45.454302 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:45.454316 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:45.456676 (+ 2360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:45.456702 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:45.457715 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:45.457723 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:45.458031 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:45.459214 (+ 1183us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:45.459248 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:45.463449 (+ 4201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:45.467817 (+ 4368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:45.467827 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:45.467846 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:45.468512 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:45.468963 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:45.468968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:45.468971 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:45.469323 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:45.470086 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:45.470094 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:45.470557 (+ 463us) client_negotiation.cc:769] Sending connection context
0926 03:06:45.470992 (+ 435us) client_negotiation.cc:240] Negotiation successful
0926 03:06:45.471539 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":1757,"threads_started":1}
I20260926 03:06:45.473306 7405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:45.444216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37543 (local address 127.0.97.60:36961)
0926 03:06:45.444800 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:45.444805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:45.457863 (+ 13058us) server_negotiation.cc:407] Connection header received
0926 03:06:45.458004 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:45.458010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:45.458094 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:45.458211 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:45.459566 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:45.460393 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:45.462037 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:45.462851 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:45.462968 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:45.467147 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:45.467154 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:45.467159 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:45.467201 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:45.470402 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:45.471291 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:45.471296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:45.471300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:45.471445 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:45.471830 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:45.471835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:45.471839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:45.472165 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:45.472304 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:45.472768 (+ 464us) server_negotiation.cc:299] Negotiation successful
0926 03:06:45.473015 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454}
I20260926 03:06:45.474200 7535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:45.444779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36961 (local address 127.0.97.1:37543)
0926 03:06:45.457736 (+ 12957us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:45.457757 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:45.457884 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:45.458409 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:45.458415 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:45.459037 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:45.459389 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:45.459400 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:45.460544 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:45.460547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:45.461879 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:45.461893 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:45.463219 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:45.463225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:45.463364 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:45.464045 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:45.464068 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:45.466911 (+ 2843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:45.470578 (+ 3667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:45.470582 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:45.470586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:45.471154 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:45.471581 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:45.471585 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:45.471588 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:45.471710 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:45.472382 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:45.472387 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:45.472521 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 03:06:45.473539 (+ 1018us) client_negotiation.cc:240] Negotiation successful
0926 03:06:45.473807 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2118,"mutex_wait_us":61,"spinlock_wait_cycles":12160,"thread_start_us":418,"threads_started":1}
I20260926 03:06:45.475636 7530 heartbeater.cc:347] Connected to a master server at 127.0.97.60:36961
Sep 26 03:06:45 dist-test-slave-hmbw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260926 03:06:45.475883 7532 heartbeater.cc:347] Connected to a master server at 127.0.97.61:38703
I20260926 03:06:45.475984 7530 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:45.476009 7532 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:45.477151 7532 heartbeater.cc:511] Master 127.0.97.61:38703 requested a full tablet report, sending...
I20260926 03:06:45.477159 7530 heartbeater.cc:511] Master 127.0.97.60:36961 requested a full tablet report, sending...
I20260926 03:06:45.477854 7538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:45.447172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.97.1:42773)
0926 03:06:45.456404 (+ 9232us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:45.456426 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:45.456558 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:45.458932 (+ 2374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:45.458937 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:45.459610 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:45.459939 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:45.459951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:45.462242 (+ 2291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:45.462247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:45.464843 (+ 2596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:45.464862 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:45.466263 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:45.466275 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:45.466460 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:45.467071 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:45.467094 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:45.469769 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:45.473046 (+ 3277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:45.473051 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:45.473055 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:45.473547 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:45.474997 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:45.475002 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:45.475006 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:45.475169 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:45.476695 (+ 1526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:45.476700 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:45.476991 (+ 291us) client_negotiation.cc:769] Sending connection context
0926 03:06:45.477145 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 03:06:45.477456 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9076,"mutex_wait_us":80,"thread_start_us":10159,"threads_started":1}
I20260926 03:06:45.477985 7540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:45.448464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42773 (local address 127.0.97.62:36297)
0926 03:06:45.458283 (+ 9819us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:45.458289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:45.458307 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:06:45.458360 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:45.458687 (+ 327us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:45.458777 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:45.458890 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:45.461026 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:45.462070 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:45.465036 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:45.465878 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:45.465999 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:45.469997 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:45.470005 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:45.470010 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:45.470052 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:45.472915 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:45.474579 (+ 1664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:45.474585 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:45.474588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:45.474739 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:45.475363 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:45.475369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:45.475372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:45.475731 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:45.475867 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:45.477415 (+ 1548us) server_negotiation.cc:299] Negotiation successful
0926 03:06:45.477639 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9690,"thread_start_us":9598,"threads_started":1}
I20260926 03:06:45.479650 7351 ts_manager.cc:194] Registered new tserver with Master: 1f35873792864b84a8ee278ba9b71a9d (127.0.97.1:43877)
I20260926 03:06:45.479655 7156 ts_manager.cc:194] Registered new tserver with Master: 1f35873792864b84a8ee278ba9b71a9d (127.0.97.1:43877)
I20260926 03:06:45.479740 7531 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36297
I20260926 03:06:45.479851 7531 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:45.480845 7531 heartbeater.cc:511] Master 127.0.97.62:36297 requested a full tablet report, sending...
I20260926 03:06:45.481948 6558 ts_manager.cc:194] Re-registered known tserver with Master: 1f35873792864b84a8ee278ba9b71a9d (127.0.97.1:43877)
I20260926 03:06:45.482812 6558 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42773
I20260926 03:06:45.486423 7539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:45.448096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56818 (local address 127.0.97.2:34799)
0926 03:06:45.458856 (+ 10760us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:45.458866 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:45.458894 (+ 28us) server_negotiation.cc:407] Connection header received
0926 03:06:45.458964 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:45.458972 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:45.459228 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:45.459404 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:45.461355 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:45.462719 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:45.465494 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:45.466711 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:45.467066 (+ 355us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:45.479356 (+ 12290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:45.479375 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:45.479393 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:45.479436 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:45.482887 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:45.483828 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:45.483837 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:45.483841 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:45.484017 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:45.484385 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:45.484390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:45.484393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:45.484786 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:45.485117 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:45.485665 (+ 548us) server_negotiation.cc:299] Negotiation successful
0926 03:06:45.486125 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10596,"thread_start_us":10436,"threads_started":1}
I20260926 03:06:45.487618 6876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:45.571063 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 6753
I20260926 03:06:45.583863 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:34799
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:36297
--webserver_port=43247
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:36297,127.0.97.61:38703
--tserver_master_addrs=127.0.97.62:36297,127.0.97.61:38703,127.0.97.60:36961 with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:06:45.836100 7300 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: true } } }
I20260926 03:06:45.836465 7300 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:45.838130 7543 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71: attempting to promote NON_VOTER 04ff140057b04287990399ccb2cc2436 to VOTER
I20260926 03:06:45.838768 7543 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: false } }
I20260926 03:06:45.840785 7361 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [term 1 LEARNER]: Refusing update from remote peer d8e4b0a3ad2449c1be9c81c740b8ef71: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 03:06:45.840826 7165 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [term 1 FOLLOWER]: Refusing update from remote peer d8e4b0a3ad2449c1be9c81c740b8ef71: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 03:06:45.841346 7543 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 03:06:45.841717 7300 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 03:06:45.844748 7543 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 04ff140057b04287990399ccb2cc2436 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: false } } }
I20260926 03:06:45.845431 7165 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 04ff140057b04287990399ccb2cc2436 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: false } } }
I20260926 03:06:45.848968 7544 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: false } } }
I20260926 03:06:45.849329 7544 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:45.849038 7300 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: false } } }
I20260926 03:06:45.849679 7300 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:45.850410 7549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: false } } }
I20260926 03:06:45.850915 7549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:06:45.859055 7544 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: false } } }
I20260926 03:06:45.859395 7544 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71 [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:45.860293 7361 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 04ff140057b04287990399ccb2cc2436 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: false } } }
I20260926 03:06:45.864357 7550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: false } } }
I20260926 03:06:45.864697 7550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 03:06:45.910549 7542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:45.910856 7542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:45.910957 7542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:45.918619 7542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:45.918753 7542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:45.918875 7542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:06:45.928987 7542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:34799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43247
--tserver_master_addrs=127.0.97.62:36297,127.0.97.61:38703,127.0.97.60:36961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:45.930292 7542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:45.931564 7542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:45.944410 7555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:45.944507 7558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:45.944471 7556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:45.945420 7542 server_base.cc:1034] running on GCE node
I20260926 03:06:45.945986 7542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:45.946933 7542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:45.948146 7542 hybrid_clock.cc:648] HybridClock initialized: now 1790392005948101 us; error 32 us; skew 500 ppm
I20260926 03:06:45.948462 7542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:45 dist-test-slave-hmbw krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392005, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:45.951625 7542 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260926 03:06:45.953081 7542 webserver.cc:460] Webserver started at http://127.0.97.2:43247/ using document root <none> and password file <none>
I20260926 03:06:45.953792 7542 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:45.953926 7542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:45.959503 7542 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 03:06:45.962178 7565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:45.963635 7542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:06:45.963810 7542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "0c1d74ffdafa4ce78d561501e3672999"
format_stamp: "Formatted at 2026-09-26 03:06:42 on dist-test-slave-hmbw"
server_key: "ce981ada0526d303926a00b631bfc057"
server_key_iv: "6764b2955ceb16a82bb78a0fb3c01fd8"
server_key_version: "encryptionkey@0"
I20260926 03:06:45.964792 7542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:45.987282 7542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:45.991390 7542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:45.991657 7542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:45.992655 7542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:06:45.994110 7542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:45.994210 7542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:45.994316 7542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:45.994383 7542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:46.026474 7542 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34799
I20260926 03:06:46.026574 7677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34799 every 8 connection(s)
I20260926 03:06:46.027920 7542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:06:46.033972 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 7542
I20260926 03:06:46.034283 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 6886
Sep 26 03:06:46 dist-test-slave-hmbw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392005, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 26 03:06:46 dist-test-slave-hmbw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392005, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 26 03:06:46 dist-test-slave-hmbw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260926 03:06:46.075912 7683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:46.031314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36629 (local address 127.0.97.60:36961)
0926 03:06:46.048170 (+ 16856us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:46.048177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:46.048214 (+ 37us) server_negotiation.cc:407] Connection header received
0926 03:06:46.048289 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:46.048295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:46.048388 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:46.048511 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:46.055582 (+ 7071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.056905 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.059917 (+ 3012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.060593 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.060724 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:46.068291 (+ 7567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:46.068299 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:46.068304 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:46.068349 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:46.071766 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:46.072875 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:46.072882 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:46.072885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:46.073045 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:46.073640 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:46.073645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:46.073648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:46.073990 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:46.074208 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:46.075211 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0926 03:06:46.075513 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16709,"thread_start_us":22470,"threads_started":1}
I20260926 03:06:46.077833 7685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:46.031881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36961 (local address 127.0.97.2:36629)
0926 03:06:46.043663 (+ 11782us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:46.043733 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:46.045491 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:46.049076 (+ 3585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:46.049092 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:46.050065 (+ 973us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:46.051087 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:46.051121 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.057206 (+ 6085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.057214 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:46.059701 (+ 2487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:46.059724 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.061043 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.061051 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:46.061376 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:46.062279 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:46.062313 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:46.067384 (+ 5071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:46.072048 (+ 4664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:46.072060 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:46.072081 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:46.072733 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:46.073192 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:46.073197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:46.073200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:46.073481 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:46.074220 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:46.074233 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:46.074752 (+ 519us) client_negotiation.cc:769] Sending connection context
0926 03:06:46.075086 (+ 334us) client_negotiation.cc:240] Negotiation successful
0926 03:06:46.075688 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11434,"spinlock_wait_cycles":66816,"thread_start_us":11198,"threads_started":1}
Sep 26 03:06:46 dist-test-slave-hmbw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392005, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:06:46.079087 7684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:46.031881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38703 (local address 127.0.97.2:41295)
0926 03:06:46.047056 (+ 15175us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:46.047077 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:46.047206 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:46.052078 (+ 4872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:46.052084 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:46.053365 (+ 1281us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:46.053730 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:46.053744 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.059977 (+ 6233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.059983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:46.061704 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:46.061721 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.064724 (+ 3003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.064732 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:46.064870 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:46.065445 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:46.065465 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:46.068520 (+ 3055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:46.075980 (+ 7460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:46.075985 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:46.075990 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:46.076732 (+ 742us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:46.077185 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:46.077190 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:46.077194 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:46.077363 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:46.078023 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:46.078029 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:46.078180 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 03:06:46.078307 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 03:06:46.078635 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15007,"mutex_wait_us":30,"thread_start_us":14692,"threads_started":1}
I20260926 03:06:46.079336 7687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:46.035405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51618 (local address 127.0.97.3:44419)
0926 03:06:46.047777 (+ 12372us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:46.047788 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:46.055784 (+ 7996us) server_negotiation.cc:407] Connection header received
0926 03:06:46.056349 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:46.056373 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:46.056746 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:46.056903 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:46.059289 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.060713 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.062377 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.063282 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.063385 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:46.070821 (+ 7436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:46.070841 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:46.070861 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:46.070916 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:46.074739 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:46.075611 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:46.075621 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:46.075626 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:46.075782 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:46.076486 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:46.076492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:46.076496 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:46.076960 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:46.077330 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:46.078686 (+ 1356us) server_negotiation.cc:299] Negotiation successful
0926 03:06:46.078976 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12169,"thread_start_us":15154,"threads_started":1}
I20260926 03:06:46.079468 7678 heartbeater.cc:347] Connected to a master server at 127.0.97.60:36961
I20260926 03:06:46.079742 7678 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:46.080343 7682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:46.031314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41295 (local address 127.0.97.61:38703)
0926 03:06:46.051683 (+ 20369us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:46.051690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:46.051709 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:06:46.051778 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:46.051784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:46.051882 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:46.053929 (+ 2047us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:46.054108 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.055343 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.062328 (+ 6985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.063051 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.065203 (+ 2152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:46.068785 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:46.068792 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:46.068796 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:46.068854 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:46.075847 (+ 6993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:46.076902 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:46.076908 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:46.076912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:46.077056 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:46.077514 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:46.077519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:46.077522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:46.077880 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:46.079635 (+ 1755us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:46.079768 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 03:06:46.080014 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20227,"thread_start_us":20107,"threads_started":1}
I20260926 03:06:46.080392 7678 heartbeater.cc:511] Master 127.0.97.60:36961 requested a full tablet report, sending...
I20260926 03:06:46.081133 7680 heartbeater.cc:347] Connected to a master server at 127.0.97.61:38703
I20260926 03:06:46.081270 7680 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:46.081637 7680 heartbeater.cc:511] Master 127.0.97.61:38703 requested a full tablet report, sending...
I20260926 03:06:46.082615 7009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:46.082655 7156 ts_manager.cc:194] Registered new tserver with Master: 0c1d74ffdafa4ce78d561501e3672999 (127.0.97.2:34799)
I20260926 03:06:46.082655 7351 ts_manager.cc:194] Registered new tserver with Master: 0c1d74ffdafa4ce78d561501e3672999 (127.0.97.2:34799)
I20260926 03:06:46.087102 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:46.047442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53053 (local address 127.0.97.62:36297)
0926 03:06:46.058689 (+ 11247us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:46.058695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:46.063723 (+ 5028us) server_negotiation.cc:407] Connection header received
0926 03:06:46.067371 (+ 3648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:46.067377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:46.067461 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:46.067556 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:46.069346 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.070447 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.072579 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.073235 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.073396 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:46.079650 (+ 6254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:46.079657 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:46.079662 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:46.079711 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:46.084252 (+ 4541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:46.085034 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:46.085041 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:46.085045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:46.085213 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:46.085614 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:46.085618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:46.085621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:46.085960 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:46.086093 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:46.086551 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 03:06:46.086780 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10959,"thread_start_us":11052,"threads_started":1}
I20260926 03:06:46.087157 7688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:46.047278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.97.2:53053)
0926 03:06:46.063607 (+ 16329us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:46.063623 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:46.067189 (+ 3566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:46.067589 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:46.067594 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:46.068781 (+ 1187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:46.069106 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:46.069118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.071074 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.071080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:46.072433 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:46.072448 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.073550 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.073557 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:46.073749 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:46.076084 (+ 2335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:46.076105 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:46.078908 (+ 2803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:46.084406 (+ 5498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:46.084411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:46.084416 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:46.084900 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:46.085312 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:46.085317 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:46.085321 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:46.085492 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:46.086065 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:46.086070 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:46.086253 (+ 183us) client_negotiation.cc:769] Sending connection context
0926 03:06:46.086392 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 03:06:46.086787 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16190,"thread_start_us":16034,"threads_started":1}
I20260926 03:06:46.087895 7679 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36297
I20260926 03:06:46.088086 7679 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:46.088543 7679 heartbeater.cc:511] Master 127.0.97.62:36297 requested a full tablet report, sending...
I20260926 03:06:46.089555 6558 ts_manager.cc:194] Re-registered known tserver with Master: 0c1d74ffdafa4ce78d561501e3672999 (127.0.97.2:34799)
I20260926 03:06:46.090727 6558 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53053
I20260926 03:06:46.167843 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 6886
I20260926 03:06:46.181373 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44419
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:36297
--webserver_port=42529
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:36297,127.0.97.61:38703
--tserver_master_addrs=127.0.97.62:36297,127.0.97.61:38703,127.0.97.60:36961 with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:06:46.485520 7531 heartbeater.cc:503] Master 127.0.97.62:36297 was elected leader, sending a full tablet report...
W20260926 03:06:46.504433 7691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:46.504798 7691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:46.504942 7691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:46.512342 7691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:46.512459 7691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:46.512606 7691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:06:46.522703 7691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=42529
--tserver_master_addrs=127.0.97.62:36297,127.0.97.61:38703,127.0.97.60:36961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:46.524068 7691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:46.525377 7691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:46.538851 7697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:46.538936 7698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:46.538884 7700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:46.544130 7691 server_base.cc:1034] running on GCE node
I20260926 03:06:46.544844 7691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:46.545878 7691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:46.547101 7691 hybrid_clock.cc:648] HybridClock initialized: now 1790392006547043 us; error 47 us; skew 500 ppm
I20260926 03:06:46.547456 7691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:46 dist-test-slave-hmbw krb5kdc[6513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392006, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:46.550683 7691 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260926 03:06:46.552260 7691 webserver.cc:460] Webserver started at http://127.0.97.3:42529/ using document root <none> and password file <none>
I20260926 03:06:46.552994 7691 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:46.553115 7691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:46.558882 7691 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 03:06:46.561738 7707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:46.563149 7691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:06:46.563304 7691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "4065ff7c51bd4a2499e8dd30a4e51c81"
format_stamp: "Formatted at 2026-09-26 03:06:43 on dist-test-slave-hmbw"
server_key: "97f8dba83f8b4f61ffc0e6471865d5c9"
server_key_iv: "850122895d29054f0d850ec2b9d74f18"
server_key_version: "encryptionkey@0"
I20260926 03:06:46.564262 7691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:46.575925 7691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:46.580380 7691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:46.580672 7691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:46.581718 7691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:06:46.583415 7691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:46.583515 7691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:46.583618 7691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:46.583685 7691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:46.615173 7691 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44419
I20260926 03:06:46.615232 7819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44419 every 8 connection(s)
I20260926 03:06:46.616593 7691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:06:46.620951 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 7691
Sep 26 03:06:46 dist-test-slave-hmbw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392006, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 26 03:06:46 dist-test-slave-hmbw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392006, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260926 03:06:46.649191 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:46.619525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34903 (local address 127.0.97.60:36961)
0926 03:06:46.620087 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:46.620093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:46.621100 (+ 1007us) server_negotiation.cc:407] Connection header received
0926 03:06:46.623358 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:46.623363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:46.623463 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:46.623568 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:46.625273 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.625984 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.635443 (+ 9459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.636097 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.636205 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:46.642290 (+ 6085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:46.642299 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:46.642305 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:46.642348 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:46.645580 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:46.646619 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:46.646625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:46.646628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:46.646795 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:46.647709 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:46.647714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:46.647717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:46.648046 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:46.648172 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:46.648639 (+ 467us) server_negotiation.cc:299] Negotiation successful
0926 03:06:46.648881 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":336,"threads_started":1}
I20260926 03:06:46.650225 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:46.619538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33319 (local address 127.0.97.61:38703)
0926 03:06:46.620415 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:46.620421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:46.627575 (+ 7154us) server_negotiation.cc:407] Connection header received
0926 03:06:46.627659 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:46.627665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:46.627755 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:46.627866 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:46.629273 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.630432 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.632750 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.633431 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.633543 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:46.639881 (+ 6338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:46.639890 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:46.639896 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:46.639945 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:46.644314 (+ 4369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:46.645385 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:46.645392 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:46.645396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:46.645553 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:46.646088 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:46.646093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:46.646097 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:46.646481 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:46.646645 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:46.649644 (+ 2999us) server_negotiation.cc:299] Negotiation successful
0926 03:06:46.649863 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":843,"threads_started":1}
I20260926 03:06:46.650408 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:46.620107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36961 (local address 127.0.97.3:34903)
0926 03:06:46.620941 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:46.620984 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:46.623057 (+ 2073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:46.623811 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:46.623820 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:46.624423 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:46.625105 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:46.625125 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.626134 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.626138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:46.635135 (+ 8997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:46.635150 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.636541 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.636549 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:46.636756 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:46.637747 (+ 991us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:46.637771 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:46.641164 (+ 3393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:46.645718 (+ 4554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:46.645722 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:46.645725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:46.646163 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:46.646940 (+ 777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:46.646943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:46.646946 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:46.647107 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:46.648245 (+ 1138us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:46.648249 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:46.648378 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 03:06:46.649464 (+ 1086us) client_negotiation.cc:240] Negotiation successful
0926 03:06:46.649770 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":931,"thread_start_us":345,"threads_started":1}
I20260926 03:06:46.650992 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:46.620108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38703 (local address 127.0.97.3:33319)
0926 03:06:46.627001 (+ 6893us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:46.627021 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:46.627144 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:46.628047 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:46.628052 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:46.628738 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:46.629071 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:46.629083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.630658 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.630664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:46.632557 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:46.632574 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.633841 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.633848 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:46.634262 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:46.635192 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:46.635242 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:46.638808 (+ 3566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:46.644523 (+ 5715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:46.644533 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:46.644556 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:46.645229 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:46.645686 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:46.645690 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:46.645694 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:46.645936 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:46.646718 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:46.646727 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:46.647193 (+ 466us) client_negotiation.cc:769] Sending connection context
0926 03:06:46.647538 (+ 345us) client_negotiation.cc:240] Negotiation successful
0926 03:06:46.648081 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6733,"mutex_wait_us":92,"thread_start_us":6408,"threads_started":1}
I20260926 03:06:46.652171 7820 heartbeater.cc:347] Connected to a master server at 127.0.97.60:36961
I20260926 03:06:46.652520 7820 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:46.653460 7820 heartbeater.cc:511] Master 127.0.97.60:36961 requested a full tablet report, sending...
I20260926 03:06:46.655436 7351 ts_manager.cc:194] Registered new tserver with Master: 4065ff7c51bd4a2499e8dd30a4e51c81 (127.0.97.3:44419)
I20260926 03:06:46.658078 7822 heartbeater.cc:347] Connected to a master server at 127.0.97.61:38703
I20260926 03:06:46.658216 7822 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:46.659081 7822 heartbeater.cc:511] Master 127.0.97.61:38703 requested a full tablet report, sending...
Sep 26 03:06:46 dist-test-slave-hmbw krb5kdc[6513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392006, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:06:46.660970 7156 ts_manager.cc:194] Registered new tserver with Master: 4065ff7c51bd4a2499e8dd30a4e51c81 (127.0.97.3:44419)
I20260926 03:06:46.670150 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:46.627402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53247 (local address 127.0.97.62:36297)
0926 03:06:46.647535 (+ 20133us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:46.647541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:46.651922 (+ 4381us) server_negotiation.cc:407] Connection header received
0926 03:06:46.651989 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:46.651993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:46.652073 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:46.652168 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:46.653266 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.654285 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.655997 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.656577 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.656721 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:46.662975 (+ 6254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:46.662980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:46.662983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:46.663017 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:46.666347 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:46.667277 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:46.667282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:46.667286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:46.667452 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:46.667883 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:46.667888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:46.667891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:46.668307 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:46.668440 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:46.669069 (+ 629us) server_negotiation.cc:299] Negotiation successful
0926 03:06:46.669796 (+ 727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19997,"thread_start_us":20137,"threads_started":1}
I20260926 03:06:46.671156 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:46.627310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.97.3:53247)
0926 03:06:46.651579 (+ 24269us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:46.651598 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:46.651693 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:46.652355 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:46.652360 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:46.652865 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:46.653054 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:46.653065 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.654449 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.654454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:46.655832 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:46.655847 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.656745 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.656752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:46.656968 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:46.657654 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:46.657676 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:46.662788 (+ 5112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:46.666590 (+ 3802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:46.666596 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:46.666601 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:46.667114 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:46.667595 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:46.667600 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:46.667604 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:46.667765 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:46.668608 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:46.668613 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:46.668777 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 03:06:46.670436 (+ 1659us) client_negotiation.cc:240] Negotiation successful
0926 03:06:46.670804 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24145,"thread_start_us":29684,"threads_started":1}
I20260926 03:06:46.672289 7821 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36297
I20260926 03:06:46.672452 7821 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:46.672932 7821 heartbeater.cc:511] Master 127.0.97.62:36297 requested a full tablet report, sending...
I20260926 03:06:46.674484 6558 ts_manager.cc:194] Re-registered known tserver with Master: 4065ff7c51bd4a2499e8dd30a4e51c81 (127.0.97.3:44419)
I20260926 03:06:46.675705 6558 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53247
I20260926 03:06:46.679596 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:46.660052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55998 (local address 127.0.97.62:36297)
0926 03:06:46.660560 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:46.660566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:46.661106 (+ 540us) server_negotiation.cc:407] Connection header received
0926 03:06:46.661167 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:46.661172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:46.661261 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:46.661351 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:46.663651 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.664880 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.667494 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.668095 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.668252 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:46.672110 (+ 3858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:46.672116 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:46.672121 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:46.672150 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:46.676127 (+ 3977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:46.676969 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:46.676974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:46.676978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:46.677123 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:46.677481 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:46.677486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:46.677490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:46.677816 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:46.678060 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:46.679057 (+ 997us) server_negotiation.cc:299] Negotiation successful
0926 03:06:46.679263 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":274,"threads_started":1}
I20260926 03:06:46.681468 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:46.661421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56972 (local address 127.0.97.60:36961)
0926 03:06:46.661595 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:46.661601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:46.661619 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:06:46.661677 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:46.661682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:46.661754 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:46.661852 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:46.663886 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.664900 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.667793 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.668430 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.668522 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:46.672619 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:46.672626 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:46.672631 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:46.672673 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:46.676532 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:46.677349 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:46.677356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:46.677359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:46.677510 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:46.677938 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:46.677943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:46.677946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:46.678322 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:46.678582 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:46.680916 (+ 2334us) server_negotiation.cc:299] Negotiation successful
0926 03:06:46.681142 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:06:46.682689 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:46.661477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39364 (local address 127.0.97.61:38703)
0926 03:06:46.661633 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:46.661638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:46.664110 (+ 2472us) server_negotiation.cc:407] Connection header received
0926 03:06:46.664332 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:46.664337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:46.664425 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:46.664506 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:46.667560 (+ 3054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.668669 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.671028 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.671687 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.671800 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:46.676137 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:46.676144 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:46.676148 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:46.676187 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:46.679325 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:46.680339 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:46.680343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:46.680346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:46.680497 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:46.680988 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:46.680994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:46.680997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:46.681335 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:46.681561 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:46.682111 (+ 550us) server_negotiation.cc:299] Negotiation successful
0926 03:06:46.682338 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:06:46.684723 6558 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55998:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:06:46.692203 6558 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:06:46.745374 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:46.710719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43877 (local address 127.0.0.1:37098)
0926 03:06:46.738273 (+ 27554us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:46.738300 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:46.738414 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:46.739235 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:46.739239 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:46.739262 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:46.739558 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:46.739568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.741488 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.741493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:46.743211 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:46.743225 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.744504 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.744511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:46.744684 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:46.744719 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 03:06:46.744803 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 03:06:46.744937 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27436,"mutex_wait_us":43,"thread_start_us":28958,"threads_started":1}
I20260926 03:06:46.745975 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:46.711226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37098 (local address 127.0.97.1:43877)
0926 03:06:46.737703 (+ 26477us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:46.737712 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:46.738555 (+ 843us) server_negotiation.cc:407] Connection header received
0926 03:06:46.738650 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:46.738657 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:46.738852 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:46.739006 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:46.739722 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.741332 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.743366 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.744291 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.744423 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:46.744663 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:46.745518 (+ 855us) server_negotiation.cc:299] Negotiation successful
0926 03:06:46.745713 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26316,"thread_start_us":26494,"threads_started":1}
I20260926 03:06:46.746778 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:46.710234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44419 (local address 127.0.0.1:51626)
0926 03:06:46.736401 (+ 26167us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:46.736423 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:46.736602 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:46.738043 (+ 1441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:46.738048 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:46.738074 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:46.738553 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:46.738566 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.741664 (+ 3098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.741669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:46.743316 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:46.743330 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.745807 (+ 2477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.745813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:46.746006 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:46.746040 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 03:06:46.746247 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 03:06:46.746390 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26045,"mutex_wait_us":26,"thread_start_us":25920,"threads_started":1}
I20260926 03:06:46.747355 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:46.710531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51626 (local address 127.0.97.3:44419)
0926 03:06:46.731970 (+ 21439us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:46.731981 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:46.736557 (+ 4576us) server_negotiation.cc:407] Connection header received
0926 03:06:46.736756 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:46.736763 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:46.737067 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:46.737233 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:46.738751 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.740926 (+ 2175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.743498 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.744971 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.745098 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:46.745329 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:46.746790 (+ 1461us) server_negotiation.cc:299] Negotiation successful
0926 03:06:46.746975 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21193,"thread_start_us":21466,"threads_started":1}
I20260926 03:06:46.749353 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:46.709819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34799 (local address 127.0.0.1:56824)
0926 03:06:46.737756 (+ 27937us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:46.737773 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:46.737903 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:46.738393 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:46.738398 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:46.738428 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:46.738790 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:46.738803 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.740591 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.740596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:46.744515 (+ 3919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:46.744530 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.748549 (+ 4019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.748556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:46.748728 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:46.748761 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:06:46.748841 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 03:06:46.748979 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26144,"mutex_wait_us":1859,"thread_start_us":26913,"threads_started":1}
I20260926 03:06:46.749418 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:46.710148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56824 (local address 127.0.97.2:34799)
0926 03:06:46.735635 (+ 25487us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:46.735645 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:46.737917 (+ 2272us) server_negotiation.cc:407] Connection header received
0926 03:06:46.738028 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:46.738033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:46.738207 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:46.738368 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:46.738991 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.740422 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.744721 (+ 4299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.748352 (+ 3631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.748482 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:46.748718 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:46.748923 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 03:06:46.749096 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25304,"thread_start_us":25213,"threads_started":1}
I20260926 03:06:46.750355 7465 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7016df355947e79ce67d529599cb1d (DEFAULT_TABLE table=test-table [id=6fc40437058f40fb908d5d2a0b476b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:46.751049 7755 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7016df355947e79ce67d529599cb1d (DEFAULT_TABLE table=test-table [id=6fc40437058f40fb908d5d2a0b476b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:46.752017 7465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b7016df355947e79ce67d529599cb1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:46.752489 7755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b7016df355947e79ce67d529599cb1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:46.758033 7613 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7016df355947e79ce67d529599cb1d (DEFAULT_TABLE table=test-table [id=6fc40437058f40fb908d5d2a0b476b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:46.759301 7613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b7016df355947e79ce67d529599cb1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:46.769513 7853 tablet_bootstrap.cc:492] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d: Bootstrap starting.
I20260926 03:06:46.769781 7852 tablet_bootstrap.cc:492] T 3b7016df355947e79ce67d529599cb1d P 4065ff7c51bd4a2499e8dd30a4e51c81: Bootstrap starting.
I20260926 03:06:46.773046 7854 tablet_bootstrap.cc:492] T 3b7016df355947e79ce67d529599cb1d P 0c1d74ffdafa4ce78d561501e3672999: Bootstrap starting.
I20260926 03:06:46.773411 7852 tablet_bootstrap.cc:654] T 3b7016df355947e79ce67d529599cb1d P 4065ff7c51bd4a2499e8dd30a4e51c81: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:46.776453 7854 tablet_bootstrap.cc:654] T 3b7016df355947e79ce67d529599cb1d P 0c1d74ffdafa4ce78d561501e3672999: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:46.777174 7853 tablet_bootstrap.cc:654] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:46.777199 7852 log.cc:824] T 3b7016df355947e79ce67d529599cb1d P 4065ff7c51bd4a2499e8dd30a4e51c81: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:46.777403 7854 log.cc:824] T 3b7016df355947e79ce67d529599cb1d P 0c1d74ffdafa4ce78d561501e3672999: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:46.778316 7853 log.cc:824] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:46.780365 7854 tablet_bootstrap.cc:492] T 3b7016df355947e79ce67d529599cb1d P 0c1d74ffdafa4ce78d561501e3672999: No bootstrap required, opened a new log
I20260926 03:06:46.780452 7852 tablet_bootstrap.cc:492] T 3b7016df355947e79ce67d529599cb1d P 4065ff7c51bd4a2499e8dd30a4e51c81: No bootstrap required, opened a new log
I20260926 03:06:46.780661 7854 ts_tablet_manager.cc:1397] T 3b7016df355947e79ce67d529599cb1d P 0c1d74ffdafa4ce78d561501e3672999: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 03:06:46.780678 7852 ts_tablet_manager.cc:1397] T 3b7016df355947e79ce67d529599cb1d P 4065ff7c51bd4a2499e8dd30a4e51c81: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260926 03:06:46.781953 7853 tablet_bootstrap.cc:492] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d: No bootstrap required, opened a new log
I20260926 03:06:46.782205 7853 ts_tablet_manager.cc:1397] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260926 03:06:46.788971 7852 raft_consensus.cc:348] T 3b7016df355947e79ce67d529599cb1d P 4065ff7c51bd4a2499e8dd30a4e51c81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1d74ffdafa4ce78d561501e3672999" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34799 } } peers { permanent_uuid: "4065ff7c51bd4a2499e8dd30a4e51c81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44419 } } peers { permanent_uuid: "1f35873792864b84a8ee278ba9b71a9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43877 } }
I20260926 03:06:46.789366 7852 raft_consensus.cc:374] T 3b7016df355947e79ce67d529599cb1d P 4065ff7c51bd4a2499e8dd30a4e51c81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:46.789455 7852 raft_consensus.cc:729] T 3b7016df355947e79ce67d529599cb1d P 4065ff7c51bd4a2499e8dd30a4e51c81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4065ff7c51bd4a2499e8dd30a4e51c81, State: Initialized, Role: FOLLOWER
I20260926 03:06:46.790077 7852 consensus_queue.cc:260] T 3b7016df355947e79ce67d529599cb1d P 4065ff7c51bd4a2499e8dd30a4e51c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1d74ffdafa4ce78d561501e3672999" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34799 } } peers { permanent_uuid: "4065ff7c51bd4a2499e8dd30a4e51c81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44419 } } peers { permanent_uuid: "1f35873792864b84a8ee278ba9b71a9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43877 } }
I20260926 03:06:46.791862 7852 ts_tablet_manager.cc:1428] T 3b7016df355947e79ce67d529599cb1d P 4065ff7c51bd4a2499e8dd30a4e51c81: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260926 03:06:46.792552 7853 raft_consensus.cc:348] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1d74ffdafa4ce78d561501e3672999" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34799 } } peers { permanent_uuid: "4065ff7c51bd4a2499e8dd30a4e51c81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44419 } } peers { permanent_uuid: "1f35873792864b84a8ee278ba9b71a9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43877 } }
I20260926 03:06:46.792990 7853 raft_consensus.cc:374] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:46.793063 7853 raft_consensus.cc:729] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f35873792864b84a8ee278ba9b71a9d, State: Initialized, Role: FOLLOWER
I20260926 03:06:46.793697 7853 consensus_queue.cc:260] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1d74ffdafa4ce78d561501e3672999" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34799 } } peers { permanent_uuid: "4065ff7c51bd4a2499e8dd30a4e51c81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44419 } } peers { permanent_uuid: "1f35873792864b84a8ee278ba9b71a9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43877 } }
I20260926 03:06:46.795641 7821 heartbeater.cc:503] Master 127.0.97.62:36297 was elected leader, sending a full tablet report...
I20260926 03:06:46.797485 7853 ts_tablet_manager.cc:1428] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d: Time spent starting tablet: real 0.015s user 0.007s sys 0.009s
I20260926 03:06:46.800908 7854 raft_consensus.cc:348] T 3b7016df355947e79ce67d529599cb1d P 0c1d74ffdafa4ce78d561501e3672999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1d74ffdafa4ce78d561501e3672999" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34799 } } peers { permanent_uuid: "4065ff7c51bd4a2499e8dd30a4e51c81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44419 } } peers { permanent_uuid: "1f35873792864b84a8ee278ba9b71a9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43877 } }
I20260926 03:06:46.801532 7854 raft_consensus.cc:374] T 3b7016df355947e79ce67d529599cb1d P 0c1d74ffdafa4ce78d561501e3672999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:46.801645 7854 raft_consensus.cc:729] T 3b7016df355947e79ce67d529599cb1d P 0c1d74ffdafa4ce78d561501e3672999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c1d74ffdafa4ce78d561501e3672999, State: Initialized, Role: FOLLOWER
I20260926 03:06:46.802340 7854 consensus_queue.cc:260] T 3b7016df355947e79ce67d529599cb1d P 0c1d74ffdafa4ce78d561501e3672999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1d74ffdafa4ce78d561501e3672999" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34799 } } peers { permanent_uuid: "4065ff7c51bd4a2499e8dd30a4e51c81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44419 } } peers { permanent_uuid: "1f35873792864b84a8ee278ba9b71a9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43877 } }
I20260926 03:06:46.803484 7679 heartbeater.cc:503] Master 127.0.97.62:36297 was elected leader, sending a full tablet report...
I20260926 03:06:46.804292 7854 ts_tablet_manager.cc:1428] T 3b7016df355947e79ce67d529599cb1d P 0c1d74ffdafa4ce78d561501e3672999: Time spent starting tablet: real 0.024s user 0.015s sys 0.000s
W20260926 03:06:46.869592 7823 tablet.cc:2369] T 3b7016df355947e79ce67d529599cb1d P 4065ff7c51bd4a2499e8dd30a4e51c81: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:06:46.945782 7533 tablet.cc:2369] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:06:46.993420 7859 raft_consensus.cc:482] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:06:46.993718 7859 raft_consensus.cc:504] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1d74ffdafa4ce78d561501e3672999" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34799 } } peers { permanent_uuid: "4065ff7c51bd4a2499e8dd30a4e51c81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44419 } } peers { permanent_uuid: "1f35873792864b84a8ee278ba9b71a9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43877 } }
I20260926 03:06:46.995447 7859 leader_election.cc:288] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c1d74ffdafa4ce78d561501e3672999 (127.0.97.2:34799), 4065ff7c51bd4a2499e8dd30a4e51c81 (127.0.97.3:44419)
I20260926 03:06:47.001845 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:46.995830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48413 (local address 127.0.97.3:44419)
0926 03:06:46.996023 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:46.996030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:46.996855 (+ 825us) server_negotiation.cc:407] Connection header received
0926 03:06:46.996991 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:46.996994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:46.997052 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:46.997136 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:46.997705 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.998463 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:47.000179 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:47.000947 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:47.001049 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:47.001159 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:47.001493 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 03:06:47.001615 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:06:47.001856 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:46.995798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52455 (local address 127.0.97.2:34799)
0926 03:06:46.995998 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:46.996006 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:46.996753 (+ 747us) server_negotiation.cc:407] Connection header received
0926 03:06:46.996942 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:46.996948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:46.997020 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:46.997086 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:46.997618 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.998340 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:47.000190 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:47.000968 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:47.001084 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:47.001174 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:47.001499 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 03:06:47.001619 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:06:47.001981 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:46.995679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34799 (local address 127.0.97.1:52455)
0926 03:06:46.996622 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:46.996638 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:46.996791 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:46.997125 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:46.997129 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:46.997160 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:46.997473 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:46.997483 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.998484 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.998489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:47.000042 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:47.000053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:47.001213 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:47.001219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:47.001347 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:47.001377 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 03:06:47.001450 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 03:06:47.001587 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":4,"thread_start_us":546,"threads_started":1}
I20260926 03:06:47.002403 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:46.995755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44419 (local address 127.0.97.1:48413)
0926 03:06:46.996721 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:46.996739 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:46.996873 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:46.997169 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:46.997173 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:46.997196 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:46.997557 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:46.997569 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:46.998608 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:46.998612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:47.000028 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:47.000039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:47.001173 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:47.001180 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:47.001306 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:47.001366 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 03:06:47.001450 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 03:06:47.001587 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"mutex_wait_us":26,"thread_start_us":528,"threads_started":1}
I20260926 03:06:47.003284 7775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7016df355947e79ce67d529599cb1d" candidate_uuid: "1f35873792864b84a8ee278ba9b71a9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4065ff7c51bd4a2499e8dd30a4e51c81" is_pre_election: true
I20260926 03:06:47.003012 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7016df355947e79ce67d529599cb1d" candidate_uuid: "1f35873792864b84a8ee278ba9b71a9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c1d74ffdafa4ce78d561501e3672999" is_pre_election: true
I20260926 03:06:47.003782 7775 raft_consensus.cc:2454] T 3b7016df355947e79ce67d529599cb1d P 4065ff7c51bd4a2499e8dd30a4e51c81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f35873792864b84a8ee278ba9b71a9d in term 0.
I20260926 03:06:47.003844 7633 raft_consensus.cc:2454] T 3b7016df355947e79ce67d529599cb1d P 0c1d74ffdafa4ce78d561501e3672999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f35873792864b84a8ee278ba9b71a9d in term 0.
I20260926 03:06:47.004479 7418 leader_election.cc:302] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0c1d74ffdafa4ce78d561501e3672999, 1f35873792864b84a8ee278ba9b71a9d, 4065ff7c51bd4a2499e8dd30a4e51c81; no voters:
I20260926 03:06:47.004890 7859 raft_consensus.cc:2781] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:06:47.005024 7859 raft_consensus.cc:482] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:06:47.005113 7859 raft_consensus.cc:3037] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:47.007349 7859 raft_consensus.cc:504] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1d74ffdafa4ce78d561501e3672999" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34799 } } peers { permanent_uuid: "4065ff7c51bd4a2499e8dd30a4e51c81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44419 } } peers { permanent_uuid: "1f35873792864b84a8ee278ba9b71a9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43877 } }
I20260926 03:06:47.008030 7859 leader_election.cc:288] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d [CANDIDATE]: Term 1 election: Requested vote from peers 0c1d74ffdafa4ce78d561501e3672999 (127.0.97.2:34799), 4065ff7c51bd4a2499e8dd30a4e51c81 (127.0.97.3:44419)
I20260926 03:06:47.008558 7775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7016df355947e79ce67d529599cb1d" candidate_uuid: "1f35873792864b84a8ee278ba9b71a9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4065ff7c51bd4a2499e8dd30a4e51c81"
I20260926 03:06:47.008554 7633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7016df355947e79ce67d529599cb1d" candidate_uuid: "1f35873792864b84a8ee278ba9b71a9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c1d74ffdafa4ce78d561501e3672999"
I20260926 03:06:47.008742 7775 raft_consensus.cc:3037] T 3b7016df355947e79ce67d529599cb1d P 4065ff7c51bd4a2499e8dd30a4e51c81 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:47.008755 7633 raft_consensus.cc:3037] T 3b7016df355947e79ce67d529599cb1d P 0c1d74ffdafa4ce78d561501e3672999 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:47.011056 7633 raft_consensus.cc:2454] T 3b7016df355947e79ce67d529599cb1d P 0c1d74ffdafa4ce78d561501e3672999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f35873792864b84a8ee278ba9b71a9d in term 1.
I20260926 03:06:47.011053 7775 raft_consensus.cc:2454] T 3b7016df355947e79ce67d529599cb1d P 4065ff7c51bd4a2499e8dd30a4e51c81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f35873792864b84a8ee278ba9b71a9d in term 1.
I20260926 03:06:47.011543 7420 leader_election.cc:302] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f35873792864b84a8ee278ba9b71a9d, 4065ff7c51bd4a2499e8dd30a4e51c81; no voters:
I20260926 03:06:47.011890 7859 raft_consensus.cc:2781] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:47.012333 7859 raft_consensus.cc:686] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d [term 1 LEADER]: Becoming Leader. State: Replica: 1f35873792864b84a8ee278ba9b71a9d, State: Running, Role: LEADER
I20260926 03:06:47.013226 7859 consensus_queue.cc:237] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1d74ffdafa4ce78d561501e3672999" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34799 } } peers { permanent_uuid: "4065ff7c51bd4a2499e8dd30a4e51c81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44419 } } peers { permanent_uuid: "1f35873792864b84a8ee278ba9b71a9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43877 } }
I20260926 03:06:47.018847 6557 catalog_manager.cc:5495] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f35873792864b84a8ee278ba9b71a9d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "1f35873792864b84a8ee278ba9b71a9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1d74ffdafa4ce78d561501e3672999" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4065ff7c51bd4a2499e8dd30a4e51c81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f35873792864b84a8ee278ba9b71a9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43877 } health_report { overall_health: HEALTHY } } }
W20260926 03:06:47.032043 7681 tablet.cc:2369] T 3b7016df355947e79ce67d529599cb1d P 0c1d74ffdafa4ce78d561501e3672999: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:06:47.051785 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:47.046114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37104 (local address 127.0.97.1:43877)
0926 03:06:47.046314 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:47.046319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:47.046337 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:06:47.046431 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:47.046435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:47.046555 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:47.046697 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:06:47.047179 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:47.047937 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:47.049414 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:47.049824 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:47.049940 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:47.050575 (+ 635us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:06:47.051194 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:06:47.051314 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:47.051466 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 03:06:47.051552 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 03:06:47.061354 7775 raft_consensus.cc:1261] T 3b7016df355947e79ce67d529599cb1d P 4065ff7c51bd4a2499e8dd30a4e51c81 [term 1 FOLLOWER]: Refusing update from remote peer 1f35873792864b84a8ee278ba9b71a9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:47.061334 7633 raft_consensus.cc:1261] T 3b7016df355947e79ce67d529599cb1d P 0c1d74ffdafa4ce78d561501e3672999 [term 1 FOLLOWER]: Refusing update from remote peer 1f35873792864b84a8ee278ba9b71a9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:47.062346 7859 consensus_queue.cc:1035] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c1d74ffdafa4ce78d561501e3672999" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:47.062668 7863 consensus_queue.cc:1035] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4065ff7c51bd4a2499e8dd30a4e51c81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:47.081032 7866 mvcc.cc:205] Tried to move back new op lower bound from 7333445660911185920 to 7333445660730089472. Current Snapshot: MvccSnapshot[applied={T|T < 7333445660911185920}]
I20260926 03:06:47.084625 7867 mvcc.cc:205] Tried to move back new op lower bound from 7333445660911185920 to 7333445660730089472. Current Snapshot: MvccSnapshot[applied={T|T < 7333445660911185920}]
I20260926 03:06:47.086364 7868 mvcc.cc:205] Tried to move back new op lower bound from 7333445660911185920 to 7333445660730089472. Current Snapshot: MvccSnapshot[applied={T|T < 7333445660911185920}]
I20260926 03:06:47.103559 6558 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:06:47.103883 6558 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:06:47.113485 6558 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8e4b0a3ad2449c1be9c81c740b8ef71: Sending DeleteTablet for 3 replicas of tablet 3b7016df355947e79ce67d529599cb1d
I20260926 03:06:47.114691 7613 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7016df355947e79ce67d529599cb1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:47 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56824
I20260926 03:06:47.114750 7465 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7016df355947e79ce67d529599cb1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:47 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37098
I20260926 03:06:47.114753 7755 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7016df355947e79ce67d529599cb1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:47 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51626
I20260926 03:06:47.127089 7879 tablet_replica.cc:330] T 3b7016df355947e79ce67d529599cb1d P 0c1d74ffdafa4ce78d561501e3672999: stopping tablet replica
I20260926 03:06:47.128276 7879 raft_consensus.cc:2229] T 3b7016df355947e79ce67d529599cb1d P 0c1d74ffdafa4ce78d561501e3672999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:06:47.129019 7877 tablet_replica.cc:330] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d: stopping tablet replica
I20260926 03:06:47.129662 7879 raft_consensus.cc:2258] T 3b7016df355947e79ce67d529599cb1d P 0c1d74ffdafa4ce78d561501e3672999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:47.130257 7877 raft_consensus.cc:2229] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:06:47.131435 7877 raft_consensus.cc:2258] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:47.133699 7879 ts_tablet_manager.cc:1905] T 3b7016df355947e79ce67d529599cb1d P 0c1d74ffdafa4ce78d561501e3672999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:47.135886 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 6526
I20260926 03:06:47.137379 6588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:47.138185 7877 ts_tablet_manager.cc:1905] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:47.138986 7878 tablet_replica.cc:330] T 3b7016df355947e79ce67d529599cb1d P 4065ff7c51bd4a2499e8dd30a4e51c81: stopping tablet replica
I20260926 03:06:47.139480 7878 raft_consensus.cc:2229] T 3b7016df355947e79ce67d529599cb1d P 4065ff7c51bd4a2499e8dd30a4e51c81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:06:47.140055 7878 raft_consensus.cc:2258] T 3b7016df355947e79ce67d529599cb1d P 4065ff7c51bd4a2499e8dd30a4e51c81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:47.142488 7878 ts_tablet_manager.cc:1905] T 3b7016df355947e79ce67d529599cb1d P 4065ff7c51bd4a2499e8dd30a4e51c81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:47.149679 7878 ts_tablet_manager.cc:1918] T 3b7016df355947e79ce67d529599cb1d P 4065ff7c51bd4a2499e8dd30a4e51c81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:06:47.149993 7878 log.cc:1196] T 3b7016df355947e79ce67d529599cb1d P 4065ff7c51bd4a2499e8dd30a4e51c81: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/wal/wals/3b7016df355947e79ce67d529599cb1d
I20260926 03:06:47.150780 7878 ts_tablet_manager.cc:1939] T 3b7016df355947e79ce67d529599cb1d P 4065ff7c51bd4a2499e8dd30a4e51c81: Deleting consensus metadata
I20260926 03:06:47.155372 7879 ts_tablet_manager.cc:1918] T 3b7016df355947e79ce67d529599cb1d P 0c1d74ffdafa4ce78d561501e3672999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:06:47.155589 7879 log.cc:1196] T 3b7016df355947e79ce67d529599cb1d P 0c1d74ffdafa4ce78d561501e3672999: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/wal/wals/3b7016df355947e79ce67d529599cb1d
I20260926 03:06:47.156174 7879 ts_tablet_manager.cc:1939] T 3b7016df355947e79ce67d529599cb1d P 0c1d74ffdafa4ce78d561501e3672999: Deleting consensus metadata
I20260926 03:06:47.157619 7877 ts_tablet_manager.cc:1918] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:06:47.157817 7877 log.cc:1196] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-0/wal/wals/3b7016df355947e79ce67d529599cb1d
I20260926 03:06:47.158406 7877 ts_tablet_manager.cc:1939] T 3b7016df355947e79ce67d529599cb1d P 1f35873792864b84a8ee278ba9b71a9d: Deleting consensus metadata
I20260926 03:06:47.165583 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:47.122877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39366 (local address 127.0.97.61:38703)
0926 03:06:47.123086 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:47.123093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:47.135077 (+ 11984us) server_negotiation.cc:407] Connection header received
0926 03:06:47.144495 (+ 9418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:47.144503 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:47.144590 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:47.144659 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:47.149166 (+ 4507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:47.150334 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:47.154666 (+ 4332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:47.155259 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:47.155414 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:47.159572 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:47.159581 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:47.159585 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:47.159629 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:47.162666 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:47.163369 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:47.163375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:47.163379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:47.163510 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:47.163893 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:47.163899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:47.163902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:47.164190 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:47.164443 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:47.164774 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 03:06:47.164994 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 03:06:47.178756 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:47.134926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56980 (local address 127.0.97.60:36961)
0926 03:06:47.135150 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:47.135157 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:47.165430 (+ 30273us) server_negotiation.cc:407] Connection header received
0926 03:06:47.165687 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:47.165693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:47.165788 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:47.165860 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:47.167730 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:47.168732 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:47.170259 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:47.170747 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:47.170865 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:47.173430 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:47.173436 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:47.173439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:47.173474 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:47.176111 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:47.176745 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:47.176751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:47.176755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:47.176875 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:47.177220 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:47.177226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:47.177229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:47.177516 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:47.177749 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:47.178045 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 03:06:47.178303 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 03:06:47.301368 6543 catalog_manager.cc:4841] TS 1f35873792864b84a8ee278ba9b71a9d (127.0.97.1:43877): tablet 3b7016df355947e79ce67d529599cb1d (table test-table [id=6fc40437058f40fb908d5d2a0b476b2a]) successfully deleted
I20260926 03:06:47.302520 6542 catalog_manager.cc:4841] TS 0c1d74ffdafa4ce78d561501e3672999 (127.0.97.2:34799): tablet 3b7016df355947e79ce67d529599cb1d (table test-table [id=6fc40437058f40fb908d5d2a0b476b2a]) successfully deleted
I20260926 03:06:47.302863 6544 catalog_manager.cc:4841] TS 4065ff7c51bd4a2499e8dd30a4e51c81 (127.0.97.3:44419): tablet 3b7016df355947e79ce67d529599cb1d (table test-table [id=6fc40437058f40fb908d5d2a0b476b2a]) successfully deleted
I20260926 03:06:47.303171 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 6526
W20260926 03:06:47.303041 6544 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.97.3:44419, user_credentials={real_user=slave}} blocked reactor thread for 150111us
I20260926 03:06:47.346922 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:47.332399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39368 (local address 127.0.97.61:38703)
0926 03:06:47.332613 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:47.332619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:47.332748 (+ 129us) server_negotiation.cc:407] Connection header received
0926 03:06:47.332898 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:47.332903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:47.332962 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:47.333035 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:47.334786 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:47.335898 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:47.337410 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:47.337977 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:47.338100 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:47.341115 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:47.341120 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:47.341123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:47.341154 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:47.343767 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:47.344745 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:47.344751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:47.344755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:47.344879 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:47.345227 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:47.345231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:47.345235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:47.345604 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:47.345825 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:47.346204 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 03:06:47.346438 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:06:47.346931 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:47.325729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56994 (local address 127.0.97.60:36961)
0926 03:06:47.326024 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:47.326031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:47.331576 (+ 5545us) server_negotiation.cc:407] Connection header received
0926 03:06:47.331744 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:47.331750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:47.331855 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:47.331962 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:47.333954 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:47.335114 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:47.336457 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:47.337140 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:47.337275 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:47.340375 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:47.340382 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:47.340396 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:47.340438 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:47.343853 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:47.344718 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:47.344724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:47.344728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:47.344871 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:47.345483 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:47.345488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:47.345492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:47.345850 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:47.346097 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:47.346365 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 03:06:47.346632 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 03:06:48.085886 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.085075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.97.3:58191)
0926 03:06:48.085562 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.085692 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":218,"threads_started":1}
W20260926 03:06:48.086243 7821 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:36297 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
I20260926 03:06:48.090147 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.089387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.97.2:32865)
0926 03:06:48.089813 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.089949 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":203,"threads_started":1}
W20260926 03:06:48.090590 7679 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:36297 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
I20260926 03:06:48.099918 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.099227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.97.1:55245)
0926 03:06:48.099635 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.099759 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":207,"threads_started":1}
W20260926 03:06:48.100284 7531 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:36297 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
I20260926 03:06:48.614871 7891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8e4b0a3ad2449c1be9c81c740b8ef71)
I20260926 03:06:48.615139 7891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: false } }
I20260926 03:06:48.616511 7891 leader_election.cc:288] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8e4b0a3ad2449c1be9c81c740b8ef71 (127.0.97.62:36297), 04ff140057b04287990399ccb2cc2436 (127.0.97.60:36961)
I20260926 03:06:48.618636 7894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8e4b0a3ad2449c1be9c81c740b8ef71)
I20260926 03:06:48.618854 7894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: false } }
I20260926 03:06:48.619789 7894 leader_election.cc:288] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8e4b0a3ad2449c1be9c81c740b8ef71 (127.0.97.62:36297), 8c0a7c8022f64e94b309e7d5256076eb (127.0.97.61:38703)
I20260926 03:06:48.620803 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.616682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.0.1:56458)
0926 03:06:48.620311 (+ 3629us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.620495 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3497,"thread_start_us":3329,"threads_started":1}
W20260926 03:06:48.621129 7140 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
I20260926 03:06:48.622231 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.620318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.0.1:56466)
0926 03:06:48.621647 (+ 1329us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.621874 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1208,"thread_start_us":1259,"threads_started":1}
W20260926 03:06:48.623116 7335 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
I20260926 03:06:48.624872 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.624191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.0.1:56476)
0926 03:06:48.624418 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.624603 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 03:06:48.625106 7140 leader_election.cc:334] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d8e4b0a3ad2449c1be9c81c740b8ef71 (127.0.97.62:36297): Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
I20260926 03:06:48.626857 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.626369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.0.1:56488)
0926 03:06:48.626540 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.626648 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 03:06:48.627177 7335 leader_election.cc:334] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d8e4b0a3ad2449c1be9c81c740b8ef71 (127.0.97.62:36297): Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
I20260926 03:06:48.630148 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.617053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57008 (local address 127.0.97.60:36961)
0926 03:06:48.623370 (+ 6317us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:48.623377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:48.623396 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:06:48.623457 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:48.623461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:48.623525 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:48.624226 (+ 701us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:48.624452 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.625672 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.628216 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.629218 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.629326 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:48.629412 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:48.629691 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 03:06:48.629813 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6202,"thread_start_us":6096,"threads_started":1}
I20260926 03:06:48.630165 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.616787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36961 (local address 127.0.0.1:57008)
0926 03:06:48.621010 (+ 4223us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.621027 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:48.621165 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:48.623709 (+ 2544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:48.623714 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:48.623744 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:48.624073 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:48.624086 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.625827 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.625833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:48.628020 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:48.628036 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.629374 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.629380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:48.629532 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:48.629557 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 03:06:48.629649 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 03:06:48.629795 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4142,"mutex_wait_us":57,"thread_start_us":7344,"threads_started":1}
I20260926 03:06:48.630815 7361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c0a7c8022f64e94b309e7d5256076eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "04ff140057b04287990399ccb2cc2436" is_pre_election: true
I20260926 03:06:48.631266 7361 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c0a7c8022f64e94b309e7d5256076eb in term 1.
I20260926 03:06:48.631734 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.622167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38703 (local address 127.0.0.1:39384)
0926 03:06:48.622403 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.622421 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:48.622569 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:48.625797 (+ 3228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:48.625801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:48.625827 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:48.626085 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:48.626097 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.627730 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.627735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:48.629482 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:48.629494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.631002 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.631010 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:48.631188 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:48.631211 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 03:06:48.631304 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 03:06:48.631445 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":668,"threads_started":1}
I20260926 03:06:48.631848 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.624420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39384 (local address 127.0.97.61:38703)
0926 03:06:48.625496 (+ 1076us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:48.625501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:48.625523 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:06:48.625580 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:48.625585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:48.625650 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:48.625759 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:48.626244 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.627577 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.629636 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.630798 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.630927 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:48.631067 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:48.631334 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 03:06:48.631506 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":967,"thread_start_us":751,"threads_started":1}
I20260926 03:06:48.631767 7141 leader_election.cc:302] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04ff140057b04287990399ccb2cc2436, 8c0a7c8022f64e94b309e7d5256076eb; no voters: d8e4b0a3ad2449c1be9c81c740b8ef71
I20260926 03:06:48.632184 7891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 03:06:48.632284 7891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [term 1 FOLLOWER]: Starting leader election (detected failure of leader d8e4b0a3ad2449c1be9c81c740b8ef71)
I20260926 03:06:48.632373 7891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:06:48.632656 7165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04ff140057b04287990399ccb2cc2436" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8c0a7c8022f64e94b309e7d5256076eb" is_pre_election: true
I20260926 03:06:48.634593 7891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: false } }
I20260926 03:06:48.635010 7165 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 04ff140057b04287990399ccb2cc2436 in current term 2: Already voted for candidate 8c0a7c8022f64e94b309e7d5256076eb in this term.
I20260926 03:06:48.635334 7891 leader_election.cc:288] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [CANDIDATE]: Term 2 election: Requested vote from peers d8e4b0a3ad2449c1be9c81c740b8ef71 (127.0.97.62:36297), 04ff140057b04287990399ccb2cc2436 (127.0.97.60:36961)
I20260926 03:06:48.635855 7361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c0a7c8022f64e94b309e7d5256076eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "04ff140057b04287990399ccb2cc2436"
I20260926 03:06:48.636066 7361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:06:48.636149 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.635466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.0.1:56500)
0926 03:06:48.635871 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.635976 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285}
I20260926 03:06:48.636438 7335 leader_election.cc:302] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 04ff140057b04287990399ccb2cc2436; no voters: 8c0a7c8022f64e94b309e7d5256076eb, d8e4b0a3ad2449c1be9c81c740b8ef71
I20260926 03:06:48.637166 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.636807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.0.1:56516)
0926 03:06:48.636964 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.637044 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 03:06:48.637342 7140 leader_election.cc:334] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d8e4b0a3ad2449c1be9c81c740b8ef71 (127.0.97.62:36297): Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
I20260926 03:06:48.638437 7361 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c0a7c8022f64e94b309e7d5256076eb in term 2.
I20260926 03:06:48.638626 7894 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 03:06:48.638918 7141 leader_election.cc:302] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04ff140057b04287990399ccb2cc2436, 8c0a7c8022f64e94b309e7d5256076eb; no voters: d8e4b0a3ad2449c1be9c81c740b8ef71
I20260926 03:06:48.639155 7891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [term 2 FOLLOWER]: Leader election won for term 2
I20260926 03:06:48.639564 7891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [term 2 LEADER]: Becoming Leader. State: Replica: 8c0a7c8022f64e94b309e7d5256076eb, State: Running, Role: LEADER
I20260926 03:06:48.639899 7891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: false } }
I20260926 03:06:48.641963 7901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c0a7c8022f64e94b309e7d5256076eb. Latest consensus state: current_term: 2 leader_uuid: "8c0a7c8022f64e94b309e7d5256076eb" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: false } } }
I20260926 03:06:48.642459 7901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:48.643086 7903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:06:48.646565 7903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:06:48.647354 7903 catalog_manager.cc:1259] Loaded cluster ID: 6aeb71c0cc784ff4b8ab8bd34d9dd50f
I20260926 03:06:48.647436 7903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:06:48.648546 7903 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:06:48.649322 7903 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb: Loaded TSK: 0
I20260926 03:06:48.650126 7903 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:06:48.682049 7155 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:06:48.689152 7155 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:06:48.693207 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.692751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.0.1:56538)
0926 03:06:48.692939 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.693030 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 03:06:48.693326 7361 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [term 2 FOLLOWER]: Refusing update from remote peer 8c0a7c8022f64e94b309e7d5256076eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 03:06:48.694013 7901 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 03:06:48.694229 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.693836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.0.1:56540)
0926 03:06:48.694020 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.694102 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 03:06:48.694391 7140 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb -> Peer d8e4b0a3ad2449c1be9c81c740b8ef71 (127.0.97.62:36297): Couldn't send request to peer d8e4b0a3ad2449c1be9c81c740b8ef71. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 03:06:48.696390 7894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c0a7c8022f64e94b309e7d5256076eb. Latest consensus state: current_term: 2 leader_uuid: "8c0a7c8022f64e94b309e7d5256076eb" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: false } } }
I20260926 03:06:48.696636 7894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:06:48.698251 7891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "8c0a7c8022f64e94b309e7d5256076eb" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: false } } }
I20260926 03:06:48.698598 7891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:48.700134 7894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "8c0a7c8022f64e94b309e7d5256076eb" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d8e4b0a3ad2449c1be9c81c740b8ef71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36297 } } peers { permanent_uuid: "8c0a7c8022f64e94b309e7d5256076eb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38703 } attrs { promote: false } } peers { permanent_uuid: "04ff140057b04287990399ccb2cc2436" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36961 } attrs { promote: false } } }
I20260926 03:06:48.700374 7894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04ff140057b04287990399ccb2cc2436 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:06:48.713709 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.706437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43877 (local address 127.0.0.1:37110)
0926 03:06:48.706959 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.706978 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:48.707092 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:48.707778 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:48.707783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:48.707808 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:48.708012 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:48.708024 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.709408 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.709414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:48.711161 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:48.711175 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.712824 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.712831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:48.713046 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:48.713088 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 03:06:48.713155 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 03:06:48.713320 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":33}
I20260926 03:06:48.714427 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.707215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51638 (local address 127.0.97.3:44419)
0926 03:06:48.707631 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:48.707635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:48.707650 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:06:48.707703 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:48.707706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:48.707783 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:48.708008 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:48.708414 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.709446 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.711838 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.712997 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.713843 (+ 846us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:48.713959 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:48.714011 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 03:06:48.714148 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":224,"threads_started":1}
I20260926 03:06:48.715027 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.706886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44419 (local address 127.0.0.1:51638)
0926 03:06:48.707019 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.707043 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:48.707142 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:48.707940 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:48.707945 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:48.707964 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:48.708269 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:48.708280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.709596 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.709602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:48.711652 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:48.711664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.713245 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.713251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:48.713446 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:48.713479 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:06:48.713540 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 03:06:48.713677 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":40}
I20260926 03:06:48.715832 7755 tablet_service.cc:1467] Processing CreateTablet for tablet 8a50c752566044549672e557dde250f1 (DEFAULT_TABLE table=test-table [id=be12fa494f9b4c8e8f63c1fd91e8a90e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:48.716041 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.706632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37110 (local address 127.0.97.1:43877)
0926 03:06:48.707474 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:48.707480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:48.707497 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:06:48.707570 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:48.707576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:48.707649 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:48.707752 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:48.708182 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.709235 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.711358 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.712630 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.715399 (+ 2769us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:48.715512 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:48.715570 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 03:06:48.715734 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":536,"threads_started":1}
I20260926 03:06:48.716282 7755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a50c752566044549672e557dde250f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:48.716781 7465 tablet_service.cc:1467] Processing CreateTablet for tablet 8a50c752566044549672e557dde250f1 (DEFAULT_TABLE table=test-table [id=be12fa494f9b4c8e8f63c1fd91e8a90e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:48.717263 7465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a50c752566044549672e557dde250f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:48.720666 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.707228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34799 (local address 127.0.0.1:56826)
0926 03:06:48.711364 (+ 4136us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.711383 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:48.711505 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:48.711964 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:48.711969 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:48.711994 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:48.712399 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:48.712410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.713761 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.713766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:48.718047 (+ 4281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:48.718064 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.719894 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.719900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:48.720070 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:48.720099 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 03:06:48.720193 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 03:06:48.720332 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4016,"thread_start_us":3819,"threads_started":1}
I20260926 03:06:48.720664 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.707635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56826 (local address 127.0.97.2:34799)
0926 03:06:48.710938 (+ 3303us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:48.710944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:48.711480 (+ 536us) server_negotiation.cc:407] Connection header received
0926 03:06:48.711627 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:48.711632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:48.711709 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:48.711798 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:48.712583 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.713606 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.718298 (+ 4692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.719520 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.719616 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:48.719719 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:48.720241 (+ 522us) server_negotiation.cc:299] Negotiation successful
0926 03:06:48.720396 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3193,"thread_start_us":3106,"threads_started":1}
I20260926 03:06:48.721709 7613 tablet_service.cc:1467] Processing CreateTablet for tablet 8a50c752566044549672e557dde250f1 (DEFAULT_TABLE table=test-table [id=be12fa494f9b4c8e8f63c1fd91e8a90e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:48.722151 7613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a50c752566044549672e557dde250f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:48.722929 7914 tablet_bootstrap.cc:492] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81: Bootstrap starting.
I20260926 03:06:48.724990 7914 tablet_bootstrap.cc:654] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:48.727720 7914 tablet_bootstrap.cc:492] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81: No bootstrap required, opened a new log
I20260926 03:06:48.727874 7914 ts_tablet_manager.cc:1397] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 03:06:48.728724 7914 raft_consensus.cc:348] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f35873792864b84a8ee278ba9b71a9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43877 } } peers { permanent_uuid: "4065ff7c51bd4a2499e8dd30a4e51c81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44419 } } peers { permanent_uuid: "0c1d74ffdafa4ce78d561501e3672999" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34799 } }
I20260926 03:06:48.729228 7914 raft_consensus.cc:374] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:48.729337 7914 raft_consensus.cc:729] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4065ff7c51bd4a2499e8dd30a4e51c81, State: Initialized, Role: FOLLOWER
I20260926 03:06:48.730006 7914 consensus_queue.cc:260] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f35873792864b84a8ee278ba9b71a9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43877 } } peers { permanent_uuid: "4065ff7c51bd4a2499e8dd30a4e51c81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44419 } } peers { permanent_uuid: "0c1d74ffdafa4ce78d561501e3672999" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34799 } }
I20260926 03:06:48.731178 7914 ts_tablet_manager.cc:1428] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260926 03:06:48.734370 7917 tablet_bootstrap.cc:492] T 8a50c752566044549672e557dde250f1 P 0c1d74ffdafa4ce78d561501e3672999: Bootstrap starting.
I20260926 03:06:48.735944 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.731287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.97.3:44331)
0926 03:06:48.735495 (+ 4208us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.735677 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4069,"thread_start_us":3910,"threads_started":1}
I20260926 03:06:48.738001 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.736533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.97.3:47421)
0926 03:06:48.737099 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.737642 (+ 543us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":284,"threads_started":1}
I20260926 03:06:48.738224 7822 heartbeater.cc:503] Master 127.0.97.61:38703 was elected leader, sending a full tablet report...
I20260926 03:06:48.739183 7915 tablet_bootstrap.cc:492] T 8a50c752566044549672e557dde250f1 P 1f35873792864b84a8ee278ba9b71a9d: Bootstrap starting.
I20260926 03:06:48.739253 7917 tablet_bootstrap.cc:654] T 8a50c752566044549672e557dde250f1 P 0c1d74ffdafa4ce78d561501e3672999: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:48.740866 7915 tablet_bootstrap.cc:654] T 8a50c752566044549672e557dde250f1 P 1f35873792864b84a8ee278ba9b71a9d: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:48.741858 7917 tablet_bootstrap.cc:492] T 8a50c752566044549672e557dde250f1 P 0c1d74ffdafa4ce78d561501e3672999: No bootstrap required, opened a new log
I20260926 03:06:48.742018 7917 ts_tablet_manager.cc:1397] T 8a50c752566044549672e557dde250f1 P 0c1d74ffdafa4ce78d561501e3672999: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260926 03:06:48.742805 7915 tablet_bootstrap.cc:492] T 8a50c752566044549672e557dde250f1 P 1f35873792864b84a8ee278ba9b71a9d: No bootstrap required, opened a new log
I20260926 03:06:48.742727 7917 raft_consensus.cc:348] T 8a50c752566044549672e557dde250f1 P 0c1d74ffdafa4ce78d561501e3672999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f35873792864b84a8ee278ba9b71a9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43877 } } peers { permanent_uuid: "4065ff7c51bd4a2499e8dd30a4e51c81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44419 } } peers { permanent_uuid: "0c1d74ffdafa4ce78d561501e3672999" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34799 } }
I20260926 03:06:48.742913 7917 raft_consensus.cc:374] T 8a50c752566044549672e557dde250f1 P 0c1d74ffdafa4ce78d561501e3672999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:48.742933 7915 ts_tablet_manager.cc:1397] T 8a50c752566044549672e557dde250f1 P 1f35873792864b84a8ee278ba9b71a9d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 03:06:48.742996 7917 raft_consensus.cc:729] T 8a50c752566044549672e557dde250f1 P 0c1d74ffdafa4ce78d561501e3672999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c1d74ffdafa4ce78d561501e3672999, State: Initialized, Role: FOLLOWER
I20260926 03:06:48.743227 7917 consensus_queue.cc:260] T 8a50c752566044549672e557dde250f1 P 0c1d74ffdafa4ce78d561501e3672999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f35873792864b84a8ee278ba9b71a9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43877 } } peers { permanent_uuid: "4065ff7c51bd4a2499e8dd30a4e51c81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44419 } } peers { permanent_uuid: "0c1d74ffdafa4ce78d561501e3672999" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34799 } }
I20260926 03:06:48.743532 7915 raft_consensus.cc:348] T 8a50c752566044549672e557dde250f1 P 1f35873792864b84a8ee278ba9b71a9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f35873792864b84a8ee278ba9b71a9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43877 } } peers { permanent_uuid: "4065ff7c51bd4a2499e8dd30a4e51c81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44419 } } peers { permanent_uuid: "0c1d74ffdafa4ce78d561501e3672999" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34799 } }
I20260926 03:06:48.743701 7915 raft_consensus.cc:374] T 8a50c752566044549672e557dde250f1 P 1f35873792864b84a8ee278ba9b71a9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:48.743791 7915 raft_consensus.cc:729] T 8a50c752566044549672e557dde250f1 P 1f35873792864b84a8ee278ba9b71a9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f35873792864b84a8ee278ba9b71a9d, State: Initialized, Role: FOLLOWER
I20260926 03:06:48.743942 7917 ts_tablet_manager.cc:1428] T 8a50c752566044549672e557dde250f1 P 0c1d74ffdafa4ce78d561501e3672999: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:06:48.743985 7915 consensus_queue.cc:260] T 8a50c752566044549672e557dde250f1 P 1f35873792864b84a8ee278ba9b71a9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f35873792864b84a8ee278ba9b71a9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43877 } } peers { permanent_uuid: "4065ff7c51bd4a2499e8dd30a4e51c81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44419 } } peers { permanent_uuid: "0c1d74ffdafa4ce78d561501e3672999" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34799 } }
I20260926 03:06:48.745632 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.744698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.97.2:55915)
0926 03:06:48.745219 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.745354 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":268,"threads_started":1}
I20260926 03:06:48.745702 7915 ts_tablet_manager.cc:1428] T 8a50c752566044549672e557dde250f1 P 1f35873792864b84a8ee278ba9b71a9d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 03:06:48.746973 7926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.746262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.97.1:32783)
0926 03:06:48.746697 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.746780 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"threads_started":1}
I20260926 03:06:48.747540 7680 heartbeater.cc:503] Master 127.0.97.61:38703 was elected leader, sending a full tablet report...
I20260926 03:06:48.749419 7532 heartbeater.cc:503] Master 127.0.97.61:38703 was elected leader, sending a full tablet report...
I20260926 03:06:48.749840 7926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.748663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.97.1:34335)
0926 03:06:48.748994 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.749076 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260926 03:06:48.875370 7918 raft_consensus.cc:482] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:06:48.875579 7918 raft_consensus.cc:504] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f35873792864b84a8ee278ba9b71a9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43877 } } peers { permanent_uuid: "4065ff7c51bd4a2499e8dd30a4e51c81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44419 } } peers { permanent_uuid: "0c1d74ffdafa4ce78d561501e3672999" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34799 } }
I20260926 03:06:48.877156 7918 leader_election.cc:288] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f35873792864b84a8ee278ba9b71a9d (127.0.97.1:43877), 0c1d74ffdafa4ce78d561501e3672999 (127.0.97.2:34799)
I20260926 03:06:48.882783 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.877532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:32985 (local address 127.0.97.1:43877)
0926 03:06:48.877727 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:48.877732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:48.877765 (+ 33us) server_negotiation.cc:407] Connection header received
0926 03:06:48.877961 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:48.877965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:48.878019 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:48.878091 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:48.878659 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.879342 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.881062 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.881826 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.881934 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:48.882034 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:48.882371 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 03:06:48.882516 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 03:06:48.882787 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.877553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50177 (local address 127.0.97.2:34799)
0926 03:06:48.877738 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:48.877743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:48.877766 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:06:48.877958 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:48.877962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:48.878021 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:48.878094 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:48.878641 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.879358 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.881063 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.881827 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.881954 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:48.882047 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:48.882401 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 03:06:48.882551 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:06:48.882824 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.877403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43877 (local address 127.0.97.3:32985)
0926 03:06:48.877660 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.877679 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:48.877824 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:48.878127 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:48.878132 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:48.878167 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:48.878517 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:48.878529 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.879470 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.879476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:48.880924 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:48.880935 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.882034 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.882041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:48.882213 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:48.882239 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 03:06:48.882326 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 03:06:48.882517 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":126,"spinlock_wait_cycles":1920}
I20260926 03:06:48.882827 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.877421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34799 (local address 127.0.97.3:50177)
0926 03:06:48.877684 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.877702 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:48.877825 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:48.878128 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:48.878133 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:48.878174 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:48.878468 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:48.878480 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.879492 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.879496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:48.880918 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:48.880929 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.882048 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.882054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:48.882245 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:48.882272 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 03:06:48.882354 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 03:06:48.882530 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":53,"spinlock_wait_cycles":2304}
I20260926 03:06:48.883587 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a50c752566044549672e557dde250f1" candidate_uuid: "4065ff7c51bd4a2499e8dd30a4e51c81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c1d74ffdafa4ce78d561501e3672999" is_pre_election: true
I20260926 03:06:48.883585 7485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a50c752566044549672e557dde250f1" candidate_uuid: "4065ff7c51bd4a2499e8dd30a4e51c81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f35873792864b84a8ee278ba9b71a9d" is_pre_election: true
I20260926 03:06:48.883901 7632 raft_consensus.cc:2454] T 8a50c752566044549672e557dde250f1 P 0c1d74ffdafa4ce78d561501e3672999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4065ff7c51bd4a2499e8dd30a4e51c81 in term 0.
I20260926 03:06:48.883968 7485 raft_consensus.cc:2454] T 8a50c752566044549672e557dde250f1 P 1f35873792864b84a8ee278ba9b71a9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4065ff7c51bd4a2499e8dd30a4e51c81 in term 0.
I20260926 03:06:48.884474 7709 leader_election.cc:302] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0c1d74ffdafa4ce78d561501e3672999, 1f35873792864b84a8ee278ba9b71a9d, 4065ff7c51bd4a2499e8dd30a4e51c81; no voters:
I20260926 03:06:48.884930 7918 raft_consensus.cc:2781] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:06:48.885032 7918 raft_consensus.cc:482] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:06:48.885119 7918 raft_consensus.cc:3037] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:48.887305 7918 raft_consensus.cc:504] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f35873792864b84a8ee278ba9b71a9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43877 } } peers { permanent_uuid: "4065ff7c51bd4a2499e8dd30a4e51c81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44419 } } peers { permanent_uuid: "0c1d74ffdafa4ce78d561501e3672999" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34799 } }
I20260926 03:06:48.887878 7918 leader_election.cc:288] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81 [CANDIDATE]: Term 1 election: Requested vote from peers 1f35873792864b84a8ee278ba9b71a9d (127.0.97.1:43877), 0c1d74ffdafa4ce78d561501e3672999 (127.0.97.2:34799)
I20260926 03:06:48.888381 7485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a50c752566044549672e557dde250f1" candidate_uuid: "4065ff7c51bd4a2499e8dd30a4e51c81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f35873792864b84a8ee278ba9b71a9d"
I20260926 03:06:48.888410 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a50c752566044549672e557dde250f1" candidate_uuid: "4065ff7c51bd4a2499e8dd30a4e51c81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c1d74ffdafa4ce78d561501e3672999"
I20260926 03:06:48.888561 7485 raft_consensus.cc:3037] T 8a50c752566044549672e557dde250f1 P 1f35873792864b84a8ee278ba9b71a9d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:48.888583 7632 raft_consensus.cc:3037] T 8a50c752566044549672e557dde250f1 P 0c1d74ffdafa4ce78d561501e3672999 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:48.890550 7632 raft_consensus.cc:2454] T 8a50c752566044549672e557dde250f1 P 0c1d74ffdafa4ce78d561501e3672999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4065ff7c51bd4a2499e8dd30a4e51c81 in term 1.
I20260926 03:06:48.890573 7485 raft_consensus.cc:2454] T 8a50c752566044549672e557dde250f1 P 1f35873792864b84a8ee278ba9b71a9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4065ff7c51bd4a2499e8dd30a4e51c81 in term 1.
I20260926 03:06:48.890971 7708 leader_election.cc:302] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c1d74ffdafa4ce78d561501e3672999, 4065ff7c51bd4a2499e8dd30a4e51c81; no voters:
I20260926 03:06:48.891260 7918 raft_consensus.cc:2781] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:48.891654 7918 raft_consensus.cc:686] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81 [term 1 LEADER]: Becoming Leader. State: Replica: 4065ff7c51bd4a2499e8dd30a4e51c81, State: Running, Role: LEADER
I20260926 03:06:48.892112 7918 consensus_queue.cc:237] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f35873792864b84a8ee278ba9b71a9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43877 } } peers { permanent_uuid: "4065ff7c51bd4a2499e8dd30a4e51c81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44419 } } peers { permanent_uuid: "0c1d74ffdafa4ce78d561501e3672999" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34799 } }
I20260926 03:06:48.892618 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.892220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.97.3:48009)
0926 03:06:48.892374 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.892474 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 03:06:48.895726 7155 catalog_manager.cc:5495] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4065ff7c51bd4a2499e8dd30a4e51c81 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "4065ff7c51bd4a2499e8dd30a4e51c81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f35873792864b84a8ee278ba9b71a9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4065ff7c51bd4a2499e8dd30a4e51c81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c1d74ffdafa4ce78d561501e3672999" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34799 } health_report { overall_health: UNKNOWN } } }
I20260926 03:06:48.920631 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.915656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51640 (local address 127.0.97.3:44419)
0926 03:06:48.915862 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:48.915868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:48.915886 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:06:48.916018 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:48.916021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:48.916094 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:48.916222 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:06:48.916595 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.917297 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.918568 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.918978 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.919087 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:48.919475 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:06:48.920049 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:06:48.920194 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:48.920288 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 03:06:48.920374 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:06:48.924772 7632 raft_consensus.cc:1261] T 8a50c752566044549672e557dde250f1 P 0c1d74ffdafa4ce78d561501e3672999 [term 1 FOLLOWER]: Refusing update from remote peer 4065ff7c51bd4a2499e8dd30a4e51c81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:48.925199 7485 raft_consensus.cc:1261] T 8a50c752566044549672e557dde250f1 P 1f35873792864b84a8ee278ba9b71a9d [term 1 FOLLOWER]: Refusing update from remote peer 4065ff7c51bd4a2499e8dd30a4e51c81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:48.925515 7927 consensus_queue.cc:1035] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c1d74ffdafa4ce78d561501e3672999" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:48.925933 7918 consensus_queue.cc:1035] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f35873792864b84a8ee278ba9b71a9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:48.929165 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.928261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.97.2:37885)
0926 03:06:48.928887 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.928988 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505}
I20260926 03:06:48.929176 7926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.928638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.97.1:45855)
0926 03:06:48.928850 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.928966 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 03:06:48.936124 7926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.935651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.97.1:48543)
0926 03:06:48.935864 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.935957 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260926 03:06:48.936136 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.935651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.97.2:32787)
0926 03:06:48.935863 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.935970 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260926 03:06:48.944567 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.943551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.97.3:34231)
0926 03:06:48.944351 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.944447 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":712}
I20260926 03:06:48.946620 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.945384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.97.3:49465)
0926 03:06:48.945617 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:48.945699 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 03:06:48.953365 7156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:06:48.953685 7156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:06:48.960564 7156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c0a7c8022f64e94b309e7d5256076eb: Sending DeleteTablet for 3 replicas of tablet 8a50c752566044549672e557dde250f1
I20260926 03:06:48.961859 7613 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a50c752566044549672e557dde250f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:48 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:56826
I20260926 03:06:48.962019 7755 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a50c752566044549672e557dde250f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:48 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:51638
I20260926 03:06:48.962277 7465 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a50c752566044549672e557dde250f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:48 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:37110
I20260926 03:06:48.963986 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 7323
I20260926 03:06:48.971698 7941 tablet_replica.cc:330] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81: stopping tablet replica
I20260926 03:06:48.972047 7941 raft_consensus.cc:2229] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:06:48.972815 7941 raft_consensus.cc:2258] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:48.973490 7940 tablet_replica.cc:330] T 8a50c752566044549672e557dde250f1 P 0c1d74ffdafa4ce78d561501e3672999: stopping tablet replica
I20260926 03:06:48.973755 7940 raft_consensus.cc:2229] T 8a50c752566044549672e557dde250f1 P 0c1d74ffdafa4ce78d561501e3672999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:06:48.974133 7940 raft_consensus.cc:2258] T 8a50c752566044549672e557dde250f1 P 0c1d74ffdafa4ce78d561501e3672999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:48.975502 7940 ts_tablet_manager.cc:1905] T 8a50c752566044549672e557dde250f1 P 0c1d74ffdafa4ce78d561501e3672999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:48.982970 7940 ts_tablet_manager.cc:1918] T 8a50c752566044549672e557dde250f1 P 0c1d74ffdafa4ce78d561501e3672999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:06:48.983225 7940 log.cc:1196] T 8a50c752566044549672e557dde250f1 P 0c1d74ffdafa4ce78d561501e3672999: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-1/wal/wals/8a50c752566044549672e557dde250f1
I20260926 03:06:48.983561 7941 ts_tablet_manager.cc:1905] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:48.983991 7940 ts_tablet_manager.cc:1939] T 8a50c752566044549672e557dde250f1 P 0c1d74ffdafa4ce78d561501e3672999: Deleting consensus metadata
I20260926 03:06:48.985376 7140 catalog_manager.cc:4841] TS 0c1d74ffdafa4ce78d561501e3672999 (127.0.97.2:34799): tablet 8a50c752566044549672e557dde250f1 (table test-table [id=be12fa494f9b4c8e8f63c1fd91e8a90e]) successfully deleted
I20260926 03:06:48.988595 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:48.965062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37124 (local address 127.0.97.1:43877)
0926 03:06:48.965237 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:48.965243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:48.971090 (+ 5847us) server_negotiation.cc:407] Connection header received
0926 03:06:48.971270 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:48.971275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:48.971389 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:48.971493 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:48.973398 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.974547 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.976446 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:48.977199 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:48.977304 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:48.981419 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:48.981441 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:48.981464 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:48.981517 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:48.985479 (+ 3962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:48.986392 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:48.986400 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:48.986403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:48.986598 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:48.987008 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:48.987014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:48.987018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:48.987421 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:48.987710 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:48.988011 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 03:06:48.988222 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:06:48.989265 7525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:48.995922 7941 ts_tablet_manager.cc:1918] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:06:48.996069 7941 log.cc:1196] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391950058021-388-0/minicluster-data/ts-2/wal/wals/8a50c752566044549672e557dde250f1
I20260926 03:06:48.996467 7941 ts_tablet_manager.cc:1939] T 8a50c752566044549672e557dde250f1 P 4065ff7c51bd4a2499e8dd30a4e51c81: Deleting consensus metadata
I20260926 03:06:48.997635 7142 catalog_manager.cc:4841] TS 4065ff7c51bd4a2499e8dd30a4e51c81 (127.0.97.3:44419): tablet 8a50c752566044549672e557dde250f1 (table test-table [id=be12fa494f9b4c8e8f63c1fd91e8a90e]) successfully deleted
I20260926 03:06:49.115387 7942 tablet_replica.cc:330] T 8a50c752566044549672e557dde250f1 P 1f35873792864b84a8ee278ba9b71a9d: stopping tablet replica
I20260926 03:06:49.115630 7942 raft_consensus.cc:2229] T 8a50c752566044549672e557dde250f1 P 1f35873792864b84a8ee278ba9b71a9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:06:49.115985 7942 raft_consensus.cc:2258] T 8a50c752566044549672e557dde250f1 P 1f35873792864b84a8ee278ba9b71a9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:49.116469 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 7323
I20260926 03:06:49.135622 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:49.134958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.0.1:56550)
0926 03:06:49.135333 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:49.135441 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236}
I20260926 03:06:49.136399 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 7542
I20260926 03:06:49.137555 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:49.136987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43877 (local address 127.0.0.1:37138)
0926 03:06:49.137308 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:49.137401 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260926 03:06:49.137944 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:49.137125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36297 (local address 127.0.0.1:56562)
0926 03:06:49.137736 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:49.137822 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:36297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535}
W20260926 03:06:49.138265 7141 catalog_manager.cc:4584] TS 1f35873792864b84a8ee278ba9b71a9d (127.0.97.1:43877): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a50c752566044549672e557dde250f1: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43877: connect: Connection refused (error 111)
I20260926 03:06:49.156632 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:49.137699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56836 (local address 127.0.97.2:34799)
0926 03:06:49.138020 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:49.138026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:49.138046 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:06:49.138590 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:49.138596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:49.138686 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:49.138813 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:49.140351 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:49.141705 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:49.144131 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:49.144763 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:49.144921 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:49.148735 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:49.148756 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:49.148777 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:49.148825 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:49.152645 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:49.153514 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:49.153524 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:49.153529 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:49.153680 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:49.154074 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:49.154079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:49.154082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:49.154563 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:49.154858 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:49.156038 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0926 03:06:49.156260 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:06:49.157398 7673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:49.199098 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:49.198442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43877 (local address 127.0.0.1:37150)
0926 03:06:49.198747 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:49.198864 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206}
I20260926 03:06:49.200070 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:49.199758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43877 (local address 127.0.0.1:37160)
0926 03:06:49.199895 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:49.199961 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 03:06:49.268543 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:49.268056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43877 (local address 127.0.0.1:37176)
0926 03:06:49.268248 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:49.268365 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 03:06:49.269683 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:49.269308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43877 (local address 127.0.0.1:37184)
0926 03:06:49.269502 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:49.269572 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 03:06:49.309098 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 7542
I20260926 03:06:49.323740 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 7691
I20260926 03:06:49.337189 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:49.324564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51656 (local address 127.0.97.3:44419)
0926 03:06:49.324767 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:49.324773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:49.324793 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:06:49.324965 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:49.324971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:49.325058 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:49.325141 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:49.326391 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:49.327288 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:49.328460 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:49.328892 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:49.329008 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:49.331520 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:49.331534 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:49.331551 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:49.331598 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:49.334478 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:49.335175 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:49.335182 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:49.335185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:49.335370 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:49.335728 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:49.335732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:49.335734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:49.336071 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:49.336387 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:49.336603 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 03:06:49.336835 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 03:06:49.337903 7815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:49.371321 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:49.370829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43877 (local address 127.0.0.1:37192)
0926 03:06:49.371031 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:49.371151 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 03:06:49.372344 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:49.372017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43877 (local address 127.0.0.1:37194)
0926 03:06:49.372171 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:49.372238 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 03:06:49.462167 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 7691
I20260926 03:06:49.478416 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 7019
I20260926 03:06:49.479449 7186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:49.598265 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 7019
I20260926 03:06:49.598318 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:49.597766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43877 (local address 127.0.0.1:37208)
0926 03:06:49.597977 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:49.598078 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 03:06:49.615531 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 7119
I20260926 03:06:49.616587 7381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:49.729666 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 7119
2026-09-26T03:06:49Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9591 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:06:49 dist-test-slave-hmbw krb5kdc[7951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:06:49 dist-test-slave-hmbw krb5kdc[7951](info): set up 2 sockets
Sep 26 03:06:49 dist-test-slave-hmbw krb5kdc[7951](info): commencing operation
krb5kdc: starting...
W20260926 03:06:50.751956 388 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.008s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:06:50 dist-test-slave-hmbw krb5kdc[7951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:06:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:06:50Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260926 03:06:50.977661 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46753
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:51.300730 7964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:51.301071 7964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:51.301196 7964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:51.308633 7964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:06:51.308779 7964 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 03:06:51.308840 7964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:51.308923 7964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:51.308987 7964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:06:51.309070 7964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:06:51.319311 7964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46753
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:51.320688 7964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:51.322034 7964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:51.334133 7970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:51.334188 7972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:51.334159 7969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:51.335132 7964 server_base.cc:1034] running on GCE node
I20260926 03:06:51.336006 7964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:51.337520 7964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:51.338740 7964 hybrid_clock.cc:648] HybridClock initialized: now 1790392011338672 us; error 52 us; skew 500 ppm
I20260926 03:06:51.339114 7964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:51 dist-test-slave-hmbw krb5kdc[7951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392011, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:51.342228 7964 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260926 03:06:51.343845 7964 webserver.cc:460] Webserver started at http://127.0.97.62:40361/ using document root <none> and password file <none>
I20260926 03:06:51.344615 7964 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:51.344723 7964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:51.345058 7964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:51.348264 7964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "665ebc9724d34582a9f1ba3490f657ab"
format_stamp: "Formatted at 2026-09-26 03:06:51 on dist-test-slave-hmbw"
server_key: "585921dbd159345bbd767c5b88559d0e"
server_key_iv: "e8eeab6c31a8ac92834ada4251779fc5"
server_key_version: "encryptionkey@0"
I20260926 03:06:51.349102 7964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "665ebc9724d34582a9f1ba3490f657ab"
format_stamp: "Formatted at 2026-09-26 03:06:51 on dist-test-slave-hmbw"
server_key: "585921dbd159345bbd767c5b88559d0e"
server_key_iv: "e8eeab6c31a8ac92834ada4251779fc5"
server_key_version: "encryptionkey@0"
I20260926 03:06:51.354579 7964 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 03:06:51.358111 7979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:51.359634 7964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:06:51.359802 7964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "665ebc9724d34582a9f1ba3490f657ab"
format_stamp: "Formatted at 2026-09-26 03:06:51 on dist-test-slave-hmbw"
server_key: "585921dbd159345bbd767c5b88559d0e"
server_key_iv: "e8eeab6c31a8ac92834ada4251779fc5"
server_key_version: "encryptionkey@0"
I20260926 03:06:51.360010 7964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:51.374709 7964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:51.379354 7964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:51.379616 7964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:51.399618 7964 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46753
I20260926 03:06:51.399638 8040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46753 every 8 connection(s)
I20260926 03:06:51.400926 7964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:06:51.404580 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:51.408008 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 7964
I20260926 03:06:51.408216 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:06:51.408826 388 external_mini_cluster.cc:1428] Setting key 72730bf1fb731e71975c5671a27fb724
I20260926 03:06:51.417647 8041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 665ebc9724d34582a9f1ba3490f657ab: Bootstrap starting.
I20260926 03:06:51.421360 8041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 665ebc9724d34582a9f1ba3490f657ab: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:51.422377 8041 log.cc:824] T 00000000000000000000000000000000 P 665ebc9724d34582a9f1ba3490f657ab: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:06:51 dist-test-slave-hmbw krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:06:51.425411 8041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 665ebc9724d34582a9f1ba3490f657ab: No bootstrap required, opened a new log
I20260926 03:06:51.432391 8044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:51.410818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50864 (local address 127.0.97.62:46753)
0926 03:06:51.411567 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:51.411583 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:51.411633 (+ 50us) server_negotiation.cc:407] Connection header received
0926 03:06:51.413416 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:51.413461 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:51.414036 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:51.414533 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:51.415911 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:51.417898 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:51.419256 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:51.420325 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:51.420521 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:51.424615 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:51.424623 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:51.424639 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:51.424680 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:51.427867 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:51.428763 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:51.428770 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:51.428831 (+ 61us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:51.429017 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:51.429417 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:51.429421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:51.429423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:51.430075 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:51.430310 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:51.430858 (+ 548us) server_negotiation.cc:299] Negotiation successful
0926 03:06:51.431287 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":302,"threads_started":1}
I20260926 03:06:51.434103 8041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 665ebc9724d34582a9f1ba3490f657ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665ebc9724d34582a9f1ba3490f657ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46753 } }
I20260926 03:06:51.434455 8041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 665ebc9724d34582a9f1ba3490f657ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:51.434590 8041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 665ebc9724d34582a9f1ba3490f657ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 665ebc9724d34582a9f1ba3490f657ab, State: Initialized, Role: FOLLOWER
I20260926 03:06:51.435210 8041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 665ebc9724d34582a9f1ba3490f657ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665ebc9724d34582a9f1ba3490f657ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46753 } }
I20260926 03:06:51.435427 8041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 665ebc9724d34582a9f1ba3490f657ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:06:51.435549 8041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 665ebc9724d34582a9f1ba3490f657ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:06:51.435695 8041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 665ebc9724d34582a9f1ba3490f657ab [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:51.437767 8041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 665ebc9724d34582a9f1ba3490f657ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665ebc9724d34582a9f1ba3490f657ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46753 } }
I20260926 03:06:51.438247 8041 leader_election.cc:302] T 00000000000000000000000000000000 P 665ebc9724d34582a9f1ba3490f657ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 665ebc9724d34582a9f1ba3490f657ab; no voters:
I20260926 03:06:51.438832 8041 leader_election.cc:288] T 00000000000000000000000000000000 P 665ebc9724d34582a9f1ba3490f657ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:06:51.438953 8046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 665ebc9724d34582a9f1ba3490f657ab [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:51.439630 8046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 665ebc9724d34582a9f1ba3490f657ab [term 1 LEADER]: Becoming Leader. State: Replica: 665ebc9724d34582a9f1ba3490f657ab, State: Running, Role: LEADER
I20260926 03:06:51.440191 8046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 665ebc9724d34582a9f1ba3490f657ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665ebc9724d34582a9f1ba3490f657ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46753 } }
I20260926 03:06:51.440806 8041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 665ebc9724d34582a9f1ba3490f657ab [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:06:51.443011 8048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 665ebc9724d34582a9f1ba3490f657ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 665ebc9724d34582a9f1ba3490f657ab. Latest consensus state: current_term: 1 leader_uuid: "665ebc9724d34582a9f1ba3490f657ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665ebc9724d34582a9f1ba3490f657ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46753 } } }
I20260926 03:06:51.443394 8048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 665ebc9724d34582a9f1ba3490f657ab [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:51.444808 8053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:06:51.444689 8047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 665ebc9724d34582a9f1ba3490f657ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "665ebc9724d34582a9f1ba3490f657ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665ebc9724d34582a9f1ba3490f657ab" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46753 } } }
I20260926 03:06:51.445017 8047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 665ebc9724d34582a9f1ba3490f657ab [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:51.450378 8053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:06:51.459918 8053 catalog_manager.cc:1347] Generated new cluster ID: 0c4c03080eca4ca19815d6804dbb91b0
I20260926 03:06:51.460070 8053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:06:51.466857 8053 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:06:51.468824 8053 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:06:51.477759 8053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 665ebc9724d34582a9f1ba3490f657ab: Generated new TSK 0
I20260926 03:06:51.478744 8053 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260926 03:06:51.539413 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46753
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:51.872179 8067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:51.872503 8067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:51.872633 8067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:51.880088 8067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:51.880236 8067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:51.880360 8067 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:06:51.880477 8067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:06:51.890453 8067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:51.891811 8067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:51.893175 8067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:51.906172 8073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:51.906245 8075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:51.906100 8072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:51.907428 8067 server_base.cc:1034] running on GCE node
I20260926 03:06:51.908123 8067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:51.909246 8067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:51.910490 8067 hybrid_clock.cc:648] HybridClock initialized: now 1790392011910444 us; error 59 us; skew 500 ppm
I20260926 03:06:51.910849 8067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:51 dist-test-slave-hmbw krb5kdc[7951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392011, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:51.914199 8067 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260926 03:06:51.915792 8067 webserver.cc:460] Webserver started at http://127.0.97.1:35085/ using document root <none> and password file <none>
I20260926 03:06:51.916482 8067 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:51.916582 8067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:51.916919 8067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:51.919961 8067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-0/data/instance:
uuid: "6ce2bd18989b4dfd89abcb754802e625"
format_stamp: "Formatted at 2026-09-26 03:06:51 on dist-test-slave-hmbw"
server_key: "c7c0918395dca61139c592d20274aa56"
server_key_iv: "4399af0871b4a16f18df56575bad4da7"
server_key_version: "encryptionkey@0"
I20260926 03:06:51.920744 8067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6ce2bd18989b4dfd89abcb754802e625"
format_stamp: "Formatted at 2026-09-26 03:06:51 on dist-test-slave-hmbw"
server_key: "c7c0918395dca61139c592d20274aa56"
server_key_iv: "4399af0871b4a16f18df56575bad4da7"
server_key_version: "encryptionkey@0"
I20260926 03:06:51.925879 8067 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 03:06:51.929807 8082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:51.931352 8067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 03:06:51.931522 8067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "6ce2bd18989b4dfd89abcb754802e625"
format_stamp: "Formatted at 2026-09-26 03:06:51 on dist-test-slave-hmbw"
server_key: "c7c0918395dca61139c592d20274aa56"
server_key_iv: "4399af0871b4a16f18df56575bad4da7"
server_key_version: "encryptionkey@0"
I20260926 03:06:51.931740 8067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:51.955410 8067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:51.959759 8067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:51.960021 8067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:51.962842 8067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:51.962966 8067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:51.963127 8067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:51.963199 8067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:06:51 dist-test-slave-hmbw krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392011, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:06:51.985548 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:51.963324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45143 (local address 127.0.97.62:46753)
0926 03:06:51.964172 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:51.964177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:51.965233 (+ 1056us) server_negotiation.cc:407] Connection header received
0926 03:06:51.967336 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:51.967342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:51.967440 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:51.967534 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:51.969828 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:51.970854 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:51.974069 (+ 3215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:51.974733 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:51.974883 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:51.979439 (+ 4556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:51.979444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:51.979447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:51.979482 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:51.981949 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:51.982833 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:51.982840 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:51.982844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:51.983004 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:51.983559 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:51.983564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:51.983566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:51.983937 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:51.984070 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:51.984933 (+ 863us) server_negotiation.cc:299] Negotiation successful
0926 03:06:51.985183 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":632,"threads_started":1}
I20260926 03:06:51.987550 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:51.964185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46753 (local address 127.0.97.1:45143)
0926 03:06:51.965045 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:51.965094 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:51.967046 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:51.967869 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:51.967882 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:51.968671 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:51.969607 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:51.969643 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:51.971058 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:51.971068 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:51.973815 (+ 2747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:51.973842 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:51.975051 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:51.975061 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:51.975347 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:51.976136 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:51.976165 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:51.979185 (+ 3020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:51.982101 (+ 2916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:51.982111 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:51.982130 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:51.982694 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:51.983136 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:51.983142 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:51.983146 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:51.983428 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:51.984065 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:51.984073 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:51.984477 (+ 404us) client_negotiation.cc:769] Sending connection context
0926 03:06:51.984851 (+ 374us) client_negotiation.cc:240] Negotiation successful
0926 03:06:51.985468 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":374,"threads_started":1}
I20260926 03:06:52.011411 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:51.995510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60157 (local address 127.0.97.62:46753)
0926 03:06:51.995791 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:51.995799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:51.996288 (+ 489us) server_negotiation.cc:407] Connection header received
0926 03:06:51.996526 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:51.996532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:51.996621 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:51.996757 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:51.997705 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:51.998864 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:52.001219 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:52.001792 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:52.001980 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:52.005144 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:52.005151 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:52.005154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:52.005192 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:52.008655 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:52.009412 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:52.009418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:52.009422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:52.009556 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:52.009919 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:52.009924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:52.009928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:52.010207 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:52.010336 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:52.010820 (+ 484us) server_negotiation.cc:299] Negotiation successful
0926 03:06:52.011044 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:06:52.011629 8157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:51.995071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46753 (local address 127.0.97.1:60157)
0926 03:06:51.996160 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:51.996189 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:51.996383 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:51.996795 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:51.996800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:51.997265 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:51.997559 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:51.997569 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:51.999077 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:51.999087 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:52.000954 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:52.000976 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:52.002394 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:52.002404 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:52.002640 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:52.003470 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:52.003490 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:52.004772 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:52.008825 (+ 4053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:52.008835 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:52.008839 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:52.009272 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:52.009672 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:52.009677 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:52.009680 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:52.009810 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:52.010328 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:52.010333 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:52.010557 (+ 224us) client_negotiation.cc:769] Sending connection context
0926 03:06:52.010695 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 03:06:52.011093 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"thread_start_us":641,"threads_started":1}
I20260926 03:06:52.016445 8067 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32987
I20260926 03:06:52.016464 8202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32987 every 8 connection(s)
I20260926 03:06:52.017781 8067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:06:52.019833 8203 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46753
I20260926 03:06:52.020088 8203 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:52.020701 8203 heartbeater.cc:511] Master 127.0.97.62:46753 requested a full tablet report, sending...
I20260926 03:06:52.022843 7996 ts_manager.cc:194] Registered new tserver with Master: 6ce2bd18989b4dfd89abcb754802e625 (127.0.97.1:32987)
I20260926 03:06:52.023007 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 8067
I20260926 03:06:52.023181 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:06:52.023787 388 external_mini_cluster.cc:1428] Setting key edeabba9bff68c3b13efb8f8285e807c
I20260926 03:06:52.024377 7996 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45143
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260926 03:06:52.074064 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46753
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:52.403384 8207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:52.403688 8207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:52.403820 8207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:52.411183 8207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:52.411311 8207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:52.411444 8207 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:06:52.411556 8207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:06:52.422133 8207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:52.423563 8207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:52.424945 8207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:52.437958 8215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:52.437913 8213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:52.437885 8212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:52.438813 8207 server_base.cc:1034] running on GCE node
I20260926 03:06:52.439400 8207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:52.440292 8207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:52.441507 8207 hybrid_clock.cc:648] HybridClock initialized: now 1790392012441434 us; error 45 us; skew 500 ppm
I20260926 03:06:52.441826 8207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:52 dist-test-slave-hmbw krb5kdc[7951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392012, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:52.444937 8207 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260926 03:06:52.446370 8207 webserver.cc:460] Webserver started at http://127.0.97.2:45115/ using document root <none> and password file <none>
I20260926 03:06:52.447088 8207 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:52.447186 8207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:52.447492 8207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:52.450328 8207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-1/data/instance:
uuid: "d8e10deebf684e65a687a362c3ff6f92"
format_stamp: "Formatted at 2026-09-26 03:06:52 on dist-test-slave-hmbw"
server_key: "7bca3e93225954875f8e947a0a6bf2f9"
server_key_iv: "1fff799c2a86d1856aacd9ff4ac87fae"
server_key_version: "encryptionkey@0"
I20260926 03:06:52.451164 8207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d8e10deebf684e65a687a362c3ff6f92"
format_stamp: "Formatted at 2026-09-26 03:06:52 on dist-test-slave-hmbw"
server_key: "7bca3e93225954875f8e947a0a6bf2f9"
server_key_iv: "1fff799c2a86d1856aacd9ff4ac87fae"
server_key_version: "encryptionkey@0"
I20260926 03:06:52.456647 8207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:06:52.460340 8222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:52.461829 8207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:06:52.462021 8207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "d8e10deebf684e65a687a362c3ff6f92"
format_stamp: "Formatted at 2026-09-26 03:06:52 on dist-test-slave-hmbw"
server_key: "7bca3e93225954875f8e947a0a6bf2f9"
server_key_iv: "1fff799c2a86d1856aacd9ff4ac87fae"
server_key_version: "encryptionkey@0"
I20260926 03:06:52.462237 8207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:52.478816 8207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:52.483829 8207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:52.484113 8207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:52.487196 8207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:52.487301 8207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:52.487448 8207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:52.487507 8207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:06:52 dist-test-slave-hmbw krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392012, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:06:52.512686 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:52.487978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52037 (local address 127.0.97.62:46753)
0926 03:06:52.488169 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:52.488176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:52.489650 (+ 1474us) server_negotiation.cc:407] Connection header received
0926 03:06:52.492517 (+ 2867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:52.492524 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:52.492619 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:52.492757 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:52.495457 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:52.496389 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:52.498741 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:52.499268 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:52.499409 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:52.504732 (+ 5323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:52.504738 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:52.504741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:52.504781 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:52.508363 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:52.509760 (+ 1397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:52.509766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:52.509770 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:52.509908 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:52.510531 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:52.510537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:52.510541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:52.510970 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:52.511114 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:52.512075 (+ 961us) server_negotiation.cc:299] Negotiation successful
0926 03:06:52.512325 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 03:06:52.514348 8231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:52.488569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46753 (local address 127.0.97.2:52037)
0926 03:06:52.489431 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:52.489491 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:52.492104 (+ 2613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:52.493076 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:52.493091 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:52.494075 (+ 984us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:52.495186 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:52.495224 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:52.496573 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:52.496577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:52.498458 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:52.498477 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:52.499584 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:52.499592 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:52.499974 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:52.500911 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:52.500951 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:52.504425 (+ 3474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:52.508625 (+ 4200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:52.508637 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:52.508660 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:52.509593 (+ 933us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:52.510022 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:52.510027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:52.510035 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:52.510371 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:52.511106 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:52.511116 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:52.511592 (+ 476us) client_negotiation.cc:769] Sending connection context
0926 03:06:52.511970 (+ 378us) client_negotiation.cc:240] Negotiation successful
0926 03:06:52.512593 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":303,"threads_started":1}
I20260926 03:06:52.530650 8207 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44011
I20260926 03:06:52.530673 8341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44011 every 8 connection(s)
I20260926 03:06:52.532176 8207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:06:52.534639 8342 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46753
I20260926 03:06:52.535000 8342 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:52.535662 8342 heartbeater.cc:511] Master 127.0.97.62:46753 requested a full tablet report, sending...
I20260926 03:06:52.537124 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 8207
I20260926 03:06:52.537294 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:06:52.537292 7995 ts_manager.cc:194] Registered new tserver with Master: d8e10deebf684e65a687a362c3ff6f92 (127.0.97.2:44011)
I20260926 03:06:52.537842 388 external_mini_cluster.cc:1428] Setting key 51e014b908737ead75a4be502041d8d3
I20260926 03:06:52.538436 7995 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52037
I20260926 03:06:52.539917 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:52.524771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37825 (local address 127.0.97.62:46753)
0926 03:06:52.524953 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:52.524957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:52.525806 (+ 849us) server_negotiation.cc:407] Connection header received
0926 03:06:52.526022 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:52.526029 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:52.526117 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:52.526232 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:52.527663 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:52.528761 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:52.530877 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:52.531298 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:52.531400 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:52.533682 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:52.533689 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:52.533692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:52.533727 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:52.537167 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:52.537863 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:52.537869 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:52.537873 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:52.537993 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:52.538333 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:52.538342 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:52.538346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:52.538770 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:52.538944 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:52.539402 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 03:06:52.539600 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:06:52.540083 8326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:52.524848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46753 (local address 127.0.97.2:37825)
0926 03:06:52.525663 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:52.525692 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:52.525871 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:52.526295 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:52.526303 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:52.527024 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:52.527471 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:52.527487 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:52.528940 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:52.528945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:52.530676 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:52.530693 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:52.531499 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:52.531507 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:52.531700 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:52.532317 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:52.532340 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:52.533481 (+ 1141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:52.537331 (+ 3850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:52.537337 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:52.537341 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:52.537738 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:52.538102 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:52.538107 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:52.538111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:52.538234 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:52.538997 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:52.539001 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:52.539160 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 03:06:52.539286 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 03:06:52.539619 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":447,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260926 03:06:52.585353 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46753
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:52.907437 8346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:52.907721 8346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:52.907816 8346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:52.914994 8346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:52.915104 8346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:52.915189 8346 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:06:52.915266 8346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:06:52.925320 8346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:52.926578 8346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:52.927817 8346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:52.940864 8352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:52.941007 8351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:52.940835 8354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:52.941718 8346 server_base.cc:1034] running on GCE node
I20260926 03:06:52.942363 8346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:52.943419 8346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:52.944636 8346 hybrid_clock.cc:648] HybridClock initialized: now 1790392012944569 us; error 38 us; skew 500 ppm
I20260926 03:06:52.944921 8346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:52 dist-test-slave-hmbw krb5kdc[7951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392012, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:52.948115 8346 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260926 03:06:52.949648 8346 webserver.cc:460] Webserver started at http://127.0.97.3:41169/ using document root <none> and password file <none>
I20260926 03:06:52.950369 8346 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:52.950477 8346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:52.950806 8346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:52.953624 8346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-2/data/instance:
uuid: "f7a8e25854b0414e8fd86eb9fc11b327"
format_stamp: "Formatted at 2026-09-26 03:06:52 on dist-test-slave-hmbw"
server_key: "67a99aef3013d079ebb5972beb361de9"
server_key_iv: "c8c742efe8c611c478fa9a7ee1c015fc"
server_key_version: "encryptionkey@0"
I20260926 03:06:52.954424 8346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f7a8e25854b0414e8fd86eb9fc11b327"
format_stamp: "Formatted at 2026-09-26 03:06:52 on dist-test-slave-hmbw"
server_key: "67a99aef3013d079ebb5972beb361de9"
server_key_iv: "c8c742efe8c611c478fa9a7ee1c015fc"
server_key_version: "encryptionkey@0"
I20260926 03:06:52.959558 8346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:06:52.963063 8361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:52.964557 8346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:06:52.964720 8346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "f7a8e25854b0414e8fd86eb9fc11b327"
format_stamp: "Formatted at 2026-09-26 03:06:52 on dist-test-slave-hmbw"
server_key: "67a99aef3013d079ebb5972beb361de9"
server_key_iv: "c8c742efe8c611c478fa9a7ee1c015fc"
server_key_version: "encryptionkey@0"
I20260926 03:06:52.964942 8346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:52.985912 8346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:52.990473 8346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:52.990777 8346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:52.993498 8346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:52.993589 8346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:52.993731 8346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:52.993785 8346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:06:53 dist-test-slave-hmbw krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392012, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:06:53.019387 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:52.994412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39139 (local address 127.0.97.62:46753)
0926 03:06:52.994622 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:52.994628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:52.996025 (+ 1397us) server_negotiation.cc:407] Connection header received
0926 03:06:52.998458 (+ 2433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:52.998464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:52.998586 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:52.998678 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:53.000996 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.002181 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.004627 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.005153 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.005307 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:53.011363 (+ 6056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:53.011371 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:53.011377 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:53.011418 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:53.014359 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:53.015792 (+ 1433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:53.015799 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:53.015802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:53.016155 (+ 353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:53.016963 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:53.016968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:53.016972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:53.017475 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:53.017642 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:53.018760 (+ 1118us) server_negotiation.cc:299] Negotiation successful
0926 03:06:53.019035 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 03:06:53.022544 8370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:52.994944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46753 (local address 127.0.97.3:39139)
0926 03:06:52.995806 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:52.995857 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:52.998037 (+ 2180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:52.999057 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:52.999066 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:52.999833 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:53.000697 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:53.000733 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.002462 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.002470 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:53.004402 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:53.004420 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.005431 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.005440 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:53.005815 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:53.006961 (+ 1146us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:53.007027 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:53.011004 (+ 3977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:53.014617 (+ 3613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:53.014631 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:53.014656 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:53.015565 (+ 909us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:53.016331 (+ 766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:53.016338 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:53.016343 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:53.016782 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:53.017692 (+ 910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:53.017701 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:53.018234 (+ 533us) client_negotiation.cc:769] Sending connection context
0926 03:06:53.019917 (+ 1683us) client_negotiation.cc:240] Negotiation successful
0926 03:06:53.020653 (+ 736us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":338,"threads_started":1}
I20260926 03:06:53.027628 8203 heartbeater.cc:503] Master 127.0.97.62:46753 was elected leader, sending a full tablet report...
I20260926 03:06:53.045466 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:53.029786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35053 (local address 127.0.97.62:46753)
0926 03:06:53.029982 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:53.029989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:53.030636 (+ 647us) server_negotiation.cc:407] Connection header received
0926 03:06:53.030804 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:53.030809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:53.030891 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:53.030987 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:53.032262 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.032992 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.034384 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.034866 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.034980 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:53.037538 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:53.037545 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:53.037550 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:53.037597 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:53.042646 (+ 5049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:53.043421 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:53.043426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:53.043431 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:53.043604 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:53.044007 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:53.044011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:53.044013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:53.044301 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:53.044446 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:53.044872 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 03:06:53.045095 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:06:53.045657 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:53.029769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46753 (local address 127.0.97.3:35053)
0926 03:06:53.030468 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:53.030491 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:53.030685 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:53.031028 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:53.031033 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:53.031665 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:53.032074 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:53.032088 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.033152 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.033159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:53.034221 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:53.034236 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.035133 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.035142 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:53.035289 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:53.035959 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:53.035976 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:53.037215 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:53.042836 (+ 5621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:53.042842 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:53.042847 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:53.043270 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:53.043744 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:53.043749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:53.043753 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:53.043885 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:53.044418 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:53.044423 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:53.044607 (+ 184us) client_negotiation.cc:769] Sending connection context
0926 03:06:53.044729 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 03:06:53.045123 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":357,"threads_started":1}
I20260926 03:06:53.048897 8346 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34895
I20260926 03:06:53.048923 8480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34895 every 8 connection(s)
I20260926 03:06:53.050125 8346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:06:53.052260 8481 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46753
I20260926 03:06:53.052491 8481 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:53.053145 8481 heartbeater.cc:511] Master 127.0.97.62:46753 requested a full tablet report, sending...
I20260926 03:06:53.054557 7996 ts_manager.cc:194] Registered new tserver with Master: f7a8e25854b0414e8fd86eb9fc11b327 (127.0.97.3:34895)
I20260926 03:06:53.055369 7996 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39139
I20260926 03:06:53.055744 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 8346
I20260926 03:06:53.055894 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:06:53.056373 388 external_mini_cluster.cc:1428] Setting key 4d83b0c51a39fa53c19fbd01c11c37c3
I20260926 03:06:53.059722 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 03:06:53 dist-test-slave-hmbw krb5kdc[7951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392013, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 03:06:53 dist-test-slave-hmbw krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:06:53 dist-test-slave-hmbw krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:06:53.109045 8091 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:06:53.098442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50872 (local address 127.0.97.62:46753)
0926 03:06:53.098693 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:53.098700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:53.098832 (+ 132us) server_negotiation.cc:407] Connection header received
0926 03:06:53.099080 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:53.099085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:53.099154 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:53.099247 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:53.100660 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.101491 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.102904 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.103353 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.103467 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:53.108802 (+ 5335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50872: BlockingRecv error: recv got EOF from 127.0.0.1:50872 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 26 03:06:53 dist-test-slave-hmbw krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392013, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:06:53.126626 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:53.112721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50874 (local address 127.0.97.62:46753)
0926 03:06:53.112956 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:53.112963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:53.113209 (+ 246us) server_negotiation.cc:407] Connection header received
0926 03:06:53.113389 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:53.113394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:53.113490 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:53.113580 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:53.114827 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.115543 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.116785 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.117245 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.117364 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:53.121126 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:53.121132 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:53.121135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:53.121172 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:53.123863 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:53.124581 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:53.124587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:53.124590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:53.124719 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:53.125202 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:53.125206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:53.125208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:53.125540 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:53.125809 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:53.126056 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 03:06:53.126279 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 03:06:53.131028 7996 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:06:53.138140 7996 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:06:53.159754 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:53.150526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57562 (local address 127.0.97.3:34895)
0926 03:06:53.152562 (+ 2036us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:53.152571 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:53.152604 (+ 33us) server_negotiation.cc:407] Connection header received
0926 03:06:53.152698 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:53.152705 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:53.152972 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:53.153841 (+ 869us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:53.154145 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.155249 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.157586 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.158656 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.158771 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:53.159022 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:53.159267 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 03:06:53.159433 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1873,"thread_start_us":1714,"threads_started":1}
I20260926 03:06:53.159754 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:53.150303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34895 (local address 127.0.0.1:57562)
0926 03:06:53.151679 (+ 1376us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:53.151733 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:53.151922 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:53.153185 (+ 1263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:53.153192 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:53.153221 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:53.153627 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:53.153640 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.155379 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.155384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:53.157153 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:53.157166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.158831 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.158838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:53.159046 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:53.159094 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 03:06:53.159206 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 03:06:53.159405 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1076,"thread_start_us":2278,"threads_started":1}
I20260926 03:06:53.169433 8411 tablet_service.cc:1467] Processing CreateTablet for tablet b3cb982079f84137bc62b64aa706eafd (DEFAULT_TABLE table=test-table [id=da9d639648814ef491ede782bbb7bdad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:53.170184 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:53.150752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32987 (local address 127.0.0.1:49838)
0926 03:06:53.152834 (+ 2082us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:53.152851 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:53.152951 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:53.163529 (+ 10578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:53.163533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:53.163557 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:53.163900 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:53.163912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.165495 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.165500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:53.167489 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:53.167502 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.169332 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.169338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:53.169542 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:53.169577 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 03:06:53.169670 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 03:06:53.169816 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1977,"thread_start_us":680,"threads_started":1}
I20260926 03:06:53.170953 8411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3cb982079f84137bc62b64aa706eafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:53.170960 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:53.151308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49838 (local address 127.0.97.1:32987)
0926 03:06:53.162824 (+ 11516us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:53.162834 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:53.162869 (+ 35us) server_negotiation.cc:407] Connection header received
0926 03:06:53.162988 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:53.162997 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:53.163319 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:53.163550 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:53.164083 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.165367 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.167639 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.168778 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.168982 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:53.169199 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:53.170468 (+ 1269us) server_negotiation.cc:299] Negotiation successful
0926 03:06:53.170676 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11328,"thread_start_us":12398,"threads_started":1}
I20260926 03:06:53.172328 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:53.153497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44011 (local address 127.0.0.1:51228)
0926 03:06:53.159990 (+ 6493us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:53.160008 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:53.160104 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:53.164890 (+ 4786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:53.164895 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:53.164924 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:53.165139 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:53.165149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.166952 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.166957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:53.168792 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:53.168807 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.170827 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.170833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:53.171031 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:53.171800 (+ 769us) client_negotiation.cc:769] Sending connection context
0926 03:06:53.171907 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 03:06:53.172066 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6400,"thread_start_us":11741,"threads_started":1}
I20260926 03:06:53.173695 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:53.153206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51228 (local address 127.0.97.2:44011)
0926 03:06:53.164275 (+ 11069us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:53.164284 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:53.164316 (+ 32us) server_negotiation.cc:407] Connection header received
0926 03:06:53.164421 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:53.164429 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:53.164672 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:53.164887 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:53.165321 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.166770 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.168993 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.170380 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.170534 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:53.170729 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:53.173225 (+ 2496us) server_negotiation.cc:299] Negotiation successful
0926 03:06:53.173407 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10923,"thread_start_us":10789,"threads_started":1}
I20260926 03:06:53.174276 8133 tablet_service.cc:1467] Processing CreateTablet for tablet b3cb982079f84137bc62b64aa706eafd (DEFAULT_TABLE table=test-table [id=da9d639648814ef491ede782bbb7bdad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:53.175539 8133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3cb982079f84137bc62b64aa706eafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:53.177201 8272 tablet_service.cc:1467] Processing CreateTablet for tablet b3cb982079f84137bc62b64aa706eafd (DEFAULT_TABLE table=test-table [id=da9d639648814ef491ede782bbb7bdad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:53.178586 8272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3cb982079f84137bc62b64aa706eafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:53.187551 8505 tablet_bootstrap.cc:492] T b3cb982079f84137bc62b64aa706eafd P f7a8e25854b0414e8fd86eb9fc11b327: Bootstrap starting.
I20260926 03:06:53.188350 8506 tablet_bootstrap.cc:492] T b3cb982079f84137bc62b64aa706eafd P 6ce2bd18989b4dfd89abcb754802e625: Bootstrap starting.
I20260926 03:06:53.191565 8505 tablet_bootstrap.cc:654] T b3cb982079f84137bc62b64aa706eafd P f7a8e25854b0414e8fd86eb9fc11b327: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:53.192593 8505 log.cc:824] T b3cb982079f84137bc62b64aa706eafd P f7a8e25854b0414e8fd86eb9fc11b327: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:53.194219 8506 tablet_bootstrap.cc:654] T b3cb982079f84137bc62b64aa706eafd P 6ce2bd18989b4dfd89abcb754802e625: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:53.195574 8506 log.cc:824] T b3cb982079f84137bc62b64aa706eafd P 6ce2bd18989b4dfd89abcb754802e625: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:53.196364 8508 tablet_bootstrap.cc:492] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92: Bootstrap starting.
I20260926 03:06:53.197170 8505 tablet_bootstrap.cc:492] T b3cb982079f84137bc62b64aa706eafd P f7a8e25854b0414e8fd86eb9fc11b327: No bootstrap required, opened a new log
I20260926 03:06:53.197470 8505 ts_tablet_manager.cc:1397] T b3cb982079f84137bc62b64aa706eafd P f7a8e25854b0414e8fd86eb9fc11b327: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 03:06:53.199672 8506 tablet_bootstrap.cc:492] T b3cb982079f84137bc62b64aa706eafd P 6ce2bd18989b4dfd89abcb754802e625: No bootstrap required, opened a new log
I20260926 03:06:53.200047 8506 ts_tablet_manager.cc:1397] T b3cb982079f84137bc62b64aa706eafd P 6ce2bd18989b4dfd89abcb754802e625: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20260926 03:06:53.200253 8508 tablet_bootstrap.cc:654] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:53.201474 8508 log.cc:824] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:53.204766 8508 tablet_bootstrap.cc:492] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92: No bootstrap required, opened a new log
I20260926 03:06:53.205072 8508 ts_tablet_manager.cc:1397] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 03:06:53.207468 8506 raft_consensus.cc:348] T b3cb982079f84137bc62b64aa706eafd P 6ce2bd18989b4dfd89abcb754802e625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } }
I20260926 03:06:53.207510 8505 raft_consensus.cc:348] T b3cb982079f84137bc62b64aa706eafd P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } }
I20260926 03:06:53.207849 8506 raft_consensus.cc:374] T b3cb982079f84137bc62b64aa706eafd P 6ce2bd18989b4dfd89abcb754802e625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:53.207932 8505 raft_consensus.cc:374] T b3cb982079f84137bc62b64aa706eafd P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:53.207942 8506 raft_consensus.cc:729] T b3cb982079f84137bc62b64aa706eafd P 6ce2bd18989b4dfd89abcb754802e625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ce2bd18989b4dfd89abcb754802e625, State: Initialized, Role: FOLLOWER
I20260926 03:06:53.208042 8505 raft_consensus.cc:729] T b3cb982079f84137bc62b64aa706eafd P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7a8e25854b0414e8fd86eb9fc11b327, State: Initialized, Role: FOLLOWER
I20260926 03:06:53.208648 8506 consensus_queue.cc:260] T b3cb982079f84137bc62b64aa706eafd P 6ce2bd18989b4dfd89abcb754802e625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } }
I20260926 03:06:53.208679 8505 consensus_queue.cc:260] T b3cb982079f84137bc62b64aa706eafd P f7a8e25854b0414e8fd86eb9fc11b327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } }
I20260926 03:06:53.210441 8506 ts_tablet_manager.cc:1428] T b3cb982079f84137bc62b64aa706eafd P 6ce2bd18989b4dfd89abcb754802e625: Time spent starting tablet: real 0.010s user 0.002s sys 0.007s
I20260926 03:06:53.211354 8481 heartbeater.cc:503] Master 127.0.97.62:46753 was elected leader, sending a full tablet report...
I20260926 03:06:53.212059 8505 ts_tablet_manager.cc:1428] T b3cb982079f84137bc62b64aa706eafd P f7a8e25854b0414e8fd86eb9fc11b327: Time spent starting tablet: real 0.014s user 0.009s sys 0.008s
I20260926 03:06:53.215474 8508 raft_consensus.cc:348] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } }
I20260926 03:06:53.215996 8508 raft_consensus.cc:374] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:53.216143 8508 raft_consensus.cc:729] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8e10deebf684e65a687a362c3ff6f92, State: Initialized, Role: FOLLOWER
I20260926 03:06:53.216782 8508 consensus_queue.cc:260] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } }
I20260926 03:06:53.217790 8342 heartbeater.cc:503] Master 127.0.97.62:46753 was elected leader, sending a full tablet report...
I20260926 03:06:53.218236 8508 ts_tablet_manager.cc:1428] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260926 03:06:53.270156 8204 tablet.cc:2369] T b3cb982079f84137bc62b64aa706eafd P 6ce2bd18989b4dfd89abcb754802e625: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:06:53.284634 8343 tablet.cc:2369] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:06:53.302101 8482 tablet.cc:2369] T b3cb982079f84137bc62b64aa706eafd P f7a8e25854b0414e8fd86eb9fc11b327: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:06:53.322427 8513 raft_consensus.cc:482] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:06:53.322808 8513 raft_consensus.cc:504] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } }
I20260926 03:06:53.324654 8513 leader_election.cc:288] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7a8e25854b0414e8fd86eb9fc11b327 (127.0.97.3:34895), 6ce2bd18989b4dfd89abcb754802e625 (127.0.97.1:32987)
I20260926 03:06:53.333913 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:53.325203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47293 (local address 127.0.97.3:34895)
0926 03:06:53.325413 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:53.325419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:53.326288 (+ 869us) server_negotiation.cc:407] Connection header received
0926 03:06:53.326996 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:53.327000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:53.327059 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:53.327137 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:53.327812 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.328875 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.331650 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.332908 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.333057 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:53.333163 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:53.333528 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 03:06:53.333662 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:06:53.334065 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:53.325244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44807 (local address 127.0.97.1:32987)
0926 03:06:53.325419 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:53.325424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:53.326336 (+ 912us) server_negotiation.cc:407] Connection header received
0926 03:06:53.326602 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:53.326606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:53.326682 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:53.326836 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:53.327499 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.328690 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.331661 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.332892 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.333057 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:53.333163 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:53.333689 (+ 526us) server_negotiation.cc:299] Negotiation successful
0926 03:06:53.333824 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:06:53.334159 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:53.324963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32987 (local address 127.0.97.2:44807)
0926 03:06:53.326165 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:53.326190 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:53.326386 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:53.326903 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:53.326907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:53.326932 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:53.327300 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:53.327314 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.328891 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.328896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:53.331438 (+ 2542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:53.331459 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.333331 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.333336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:53.333523 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:53.333549 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 03:06:53.333631 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 03:06:53.333768 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"mutex_wait_us":88,"spinlock_wait_cycles":5632,"thread_start_us":879,"threads_started":1}
I20260926 03:06:53.334435 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:53.325013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34895 (local address 127.0.97.2:47293)
0926 03:06:53.326196 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:53.326209 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:53.326386 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:53.327218 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:53.327257 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:53.327286 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:53.327621 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:53.327631 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.329016 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.329021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:53.331438 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:53.331459 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.333161 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.333170 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:53.333326 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:53.333374 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 03:06:53.333480 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 03:06:53.333627 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1069,"mutex_wait_us":50,"spinlock_wait_cycles":11392,"thread_start_us":664,"threads_started":1}
I20260926 03:06:53.335358 8158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3cb982079f84137bc62b64aa706eafd" candidate_uuid: "d8e10deebf684e65a687a362c3ff6f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ce2bd18989b4dfd89abcb754802e625" is_pre_election: true
I20260926 03:06:53.335450 8431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3cb982079f84137bc62b64aa706eafd" candidate_uuid: "d8e10deebf684e65a687a362c3ff6f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" is_pre_election: true
I20260926 03:06:53.335973 8431 raft_consensus.cc:2454] T b3cb982079f84137bc62b64aa706eafd P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8e10deebf684e65a687a362c3ff6f92 in term 0.
I20260926 03:06:53.336038 8158 raft_consensus.cc:2454] T b3cb982079f84137bc62b64aa706eafd P 6ce2bd18989b4dfd89abcb754802e625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8e10deebf684e65a687a362c3ff6f92 in term 0.
I20260926 03:06:53.336656 8226 leader_election.cc:302] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8e10deebf684e65a687a362c3ff6f92, f7a8e25854b0414e8fd86eb9fc11b327; no voters:
I20260926 03:06:53.337088 8513 raft_consensus.cc:2781] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:06:53.337252 8513 raft_consensus.cc:482] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:06:53.337350 8513 raft_consensus.cc:3037] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:53.339601 8513 raft_consensus.cc:504] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } }
I20260926 03:06:53.340301 8513 leader_election.cc:288] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92 [CANDIDATE]: Term 1 election: Requested vote from peers f7a8e25854b0414e8fd86eb9fc11b327 (127.0.97.3:34895), 6ce2bd18989b4dfd89abcb754802e625 (127.0.97.1:32987)
I20260926 03:06:53.340809 8431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3cb982079f84137bc62b64aa706eafd" candidate_uuid: "d8e10deebf684e65a687a362c3ff6f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7a8e25854b0414e8fd86eb9fc11b327"
I20260926 03:06:53.341025 8431 raft_consensus.cc:3037] T b3cb982079f84137bc62b64aa706eafd P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:53.340974 8158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3cb982079f84137bc62b64aa706eafd" candidate_uuid: "d8e10deebf684e65a687a362c3ff6f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ce2bd18989b4dfd89abcb754802e625"
I20260926 03:06:53.341209 8158 raft_consensus.cc:3037] T b3cb982079f84137bc62b64aa706eafd P 6ce2bd18989b4dfd89abcb754802e625 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:53.343394 8431 raft_consensus.cc:2454] T b3cb982079f84137bc62b64aa706eafd P f7a8e25854b0414e8fd86eb9fc11b327 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8e10deebf684e65a687a362c3ff6f92 in term 1.
I20260926 03:06:53.343642 8158 raft_consensus.cc:2454] T b3cb982079f84137bc62b64aa706eafd P 6ce2bd18989b4dfd89abcb754802e625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8e10deebf684e65a687a362c3ff6f92 in term 1.
I20260926 03:06:53.343876 8226 leader_election.cc:302] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8e10deebf684e65a687a362c3ff6f92, f7a8e25854b0414e8fd86eb9fc11b327; no voters:
I20260926 03:06:53.344193 8513 raft_consensus.cc:2781] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:53.344630 8513 raft_consensus.cc:686] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92 [term 1 LEADER]: Becoming Leader. State: Replica: d8e10deebf684e65a687a362c3ff6f92, State: Running, Role: LEADER
I20260926 03:06:53.345096 8513 consensus_queue.cc:237] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } }
I20260926 03:06:53.350910 7996 catalog_manager.cc:5495] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8e10deebf684e65a687a362c3ff6f92 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "d8e10deebf684e65a687a362c3ff6f92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } health_report { overall_health: UNKNOWN } } }
I20260926 03:06:53.399976 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:53.394683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51238 (local address 127.0.97.2:44011)
0926 03:06:53.394881 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:53.394887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:53.394932 (+ 45us) server_negotiation.cc:407] Connection header received
0926 03:06:53.395164 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:53.395167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:53.395242 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:53.395385 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:06:53.395737 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.396498 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.397884 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.398323 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.398441 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:53.398857 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:06:53.399412 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:06:53.399549 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:53.399648 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 03:06:53.399732 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:06:53.409114 8431 raft_consensus.cc:1261] T b3cb982079f84137bc62b64aa706eafd P f7a8e25854b0414e8fd86eb9fc11b327 [term 1 FOLLOWER]: Refusing update from remote peer d8e10deebf684e65a687a362c3ff6f92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:53.409114 8158 raft_consensus.cc:1261] T b3cb982079f84137bc62b64aa706eafd P 6ce2bd18989b4dfd89abcb754802e625 [term 1 FOLLOWER]: Refusing update from remote peer d8e10deebf684e65a687a362c3ff6f92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:53.410257 8516 consensus_queue.cc:1035] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:53.410698 8513 consensus_queue.cc:1035] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:53.424230 8521 mvcc.cc:205] Tried to move back new op lower bound from 7333445686912745472 to 7333445686665601024. Current Snapshot: MvccSnapshot[applied={T|T < 7333445686912745472}]
I20260926 03:06:53.426388 8520 mvcc.cc:205] Tried to move back new op lower bound from 7333445686912745472 to 7333445686665601024. Current Snapshot: MvccSnapshot[applied={T|T < 7333445686912745472}]
I20260926 03:06:53.428838 8519 mvcc.cc:205] Tried to move back new op lower bound from 7333445686912745472 to 7333445686665601024. Current Snapshot: MvccSnapshot[applied={T|T < 7333445686912745472 or (T in {7333445686912745472})}]
I20260926 03:06:53.438023 7996 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:06:53.438268 7996 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:06:53.441543 7996 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 665ebc9724d34582a9f1ba3490f657ab: Sending DeleteTablet for 3 replicas of tablet b3cb982079f84137bc62b64aa706eafd
I20260926 03:06:53.442826 8133 tablet_service.cc:1514] Processing DeleteTablet for tablet b3cb982079f84137bc62b64aa706eafd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:53 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49838
I20260926 03:06:53.442883 8411 tablet_service.cc:1514] Processing DeleteTablet for tablet b3cb982079f84137bc62b64aa706eafd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:53 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57562
I20260926 03:06:53.443104 8272 tablet_service.cc:1514] Processing DeleteTablet for tablet b3cb982079f84137bc62b64aa706eafd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:53 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51228
I20260926 03:06:53.443722 8528 tablet_replica.cc:330] T b3cb982079f84137bc62b64aa706eafd P 6ce2bd18989b4dfd89abcb754802e625: stopping tablet replica
I20260926 03:06:53.444065 8528 raft_consensus.cc:2229] T b3cb982079f84137bc62b64aa706eafd P 6ce2bd18989b4dfd89abcb754802e625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:06:53.444511 8528 raft_consensus.cc:2258] T b3cb982079f84137bc62b64aa706eafd P 6ce2bd18989b4dfd89abcb754802e625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:53.444885 7996 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 03:06:53.445269 8529 tablet_replica.cc:330] T b3cb982079f84137bc62b64aa706eafd P f7a8e25854b0414e8fd86eb9fc11b327: stopping tablet replica
I20260926 03:06:53.445508 8530 tablet_replica.cc:330] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92: stopping tablet replica
I20260926 03:06:53.445655 8529 raft_consensus.cc:2229] T b3cb982079f84137bc62b64aa706eafd P f7a8e25854b0414e8fd86eb9fc11b327 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 03:06:53.445909 7996 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:06:53.445924 8530 raft_consensus.cc:2229] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:06:53.446156 8529 raft_consensus.cc:2258] T b3cb982079f84137bc62b64aa706eafd P f7a8e25854b0414e8fd86eb9fc11b327 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:53.446627 8530 raft_consensus.cc:2258] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:53.448427 8530 ts_tablet_manager.cc:1905] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:53.449011 8529 ts_tablet_manager.cc:1905] T b3cb982079f84137bc62b64aa706eafd P f7a8e25854b0414e8fd86eb9fc11b327: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:53.453958 8133 tablet_service.cc:1467] Processing CreateTablet for tablet a3bfb8c9387e46e695a625bb34f2a02c (DEFAULT_TABLE table=test-table [id=322e1349f211402fade0a5a441086aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:53.454265 8272 tablet_service.cc:1467] Processing CreateTablet for tablet a3bfb8c9387e46e695a625bb34f2a02c (DEFAULT_TABLE table=test-table [id=322e1349f211402fade0a5a441086aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:53.454408 8133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3bfb8c9387e46e695a625bb34f2a02c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:53.454689 8272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3bfb8c9387e46e695a625bb34f2a02c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:53.454669 8411 tablet_service.cc:1467] Processing CreateTablet for tablet a3bfb8c9387e46e695a625bb34f2a02c (DEFAULT_TABLE table=test-table [id=322e1349f211402fade0a5a441086aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:53.455063 8411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3bfb8c9387e46e695a625bb34f2a02c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:53.455642 8528 ts_tablet_manager.cc:1905] T b3cb982079f84137bc62b64aa706eafd P 6ce2bd18989b4dfd89abcb754802e625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:53.460911 8508 tablet_bootstrap.cc:492] T a3bfb8c9387e46e695a625bb34f2a02c P d8e10deebf684e65a687a362c3ff6f92: Bootstrap starting.
I20260926 03:06:53.461383 8506 tablet_bootstrap.cc:492] T a3bfb8c9387e46e695a625bb34f2a02c P 6ce2bd18989b4dfd89abcb754802e625: Bootstrap starting.
I20260926 03:06:53.462738 8508 tablet_bootstrap.cc:654] T a3bfb8c9387e46e695a625bb34f2a02c P d8e10deebf684e65a687a362c3ff6f92: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:53.462755 8528 ts_tablet_manager.cc:1918] T b3cb982079f84137bc62b64aa706eafd P 6ce2bd18989b4dfd89abcb754802e625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:06:53.462910 8528 log.cc:1196] T b3cb982079f84137bc62b64aa706eafd P 6ce2bd18989b4dfd89abcb754802e625: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-0/wal/wals/b3cb982079f84137bc62b64aa706eafd
I20260926 03:06:53.463074 8506 tablet_bootstrap.cc:654] T a3bfb8c9387e46e695a625bb34f2a02c P 6ce2bd18989b4dfd89abcb754802e625: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:53.463449 8528 ts_tablet_manager.cc:1939] T b3cb982079f84137bc62b64aa706eafd P 6ce2bd18989b4dfd89abcb754802e625: Deleting consensus metadata
I20260926 03:06:53.465343 7981 catalog_manager.cc:4841] TS 6ce2bd18989b4dfd89abcb754802e625 (127.0.97.1:32987): tablet b3cb982079f84137bc62b64aa706eafd (table test-table [id=da9d639648814ef491ede782bbb7bdad]) successfully deleted
I20260926 03:06:53.466367 8529 ts_tablet_manager.cc:1918] T b3cb982079f84137bc62b64aa706eafd P f7a8e25854b0414e8fd86eb9fc11b327: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:06:53.466581 8529 log.cc:1196] T b3cb982079f84137bc62b64aa706eafd P f7a8e25854b0414e8fd86eb9fc11b327: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-2/wal/wals/b3cb982079f84137bc62b64aa706eafd
I20260926 03:06:53.467113 8529 ts_tablet_manager.cc:1939] T b3cb982079f84137bc62b64aa706eafd P f7a8e25854b0414e8fd86eb9fc11b327: Deleting consensus metadata
I20260926 03:06:53.467661 8505 tablet_bootstrap.cc:492] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327: Bootstrap starting.
I20260926 03:06:53.469194 7983 catalog_manager.cc:4841] TS f7a8e25854b0414e8fd86eb9fc11b327 (127.0.97.3:34895): tablet b3cb982079f84137bc62b64aa706eafd (table test-table [id=da9d639648814ef491ede782bbb7bdad]) successfully deleted
I20260926 03:06:53.469286 8505 tablet_bootstrap.cc:654] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:53.472280 8530 ts_tablet_manager.cc:1918] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:06:53.472466 8530 log.cc:1196] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-1/wal/wals/b3cb982079f84137bc62b64aa706eafd
I20260926 03:06:53.472998 8530 ts_tablet_manager.cc:1939] T b3cb982079f84137bc62b64aa706eafd P d8e10deebf684e65a687a362c3ff6f92: Deleting consensus metadata
I20260926 03:06:53.475489 7983 catalog_manager.cc:4841] TS d8e10deebf684e65a687a362c3ff6f92 (127.0.97.2:44011): tablet b3cb982079f84137bc62b64aa706eafd (table test-table [id=da9d639648814ef491ede782bbb7bdad]) successfully deleted
I20260926 03:06:53.477120 8506 tablet_bootstrap.cc:492] T a3bfb8c9387e46e695a625bb34f2a02c P 6ce2bd18989b4dfd89abcb754802e625: No bootstrap required, opened a new log
I20260926 03:06:53.477286 8506 ts_tablet_manager.cc:1397] T a3bfb8c9387e46e695a625bb34f2a02c P 6ce2bd18989b4dfd89abcb754802e625: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.003s
I20260926 03:06:53.477325 8508 tablet_bootstrap.cc:492] T a3bfb8c9387e46e695a625bb34f2a02c P d8e10deebf684e65a687a362c3ff6f92: No bootstrap required, opened a new log
I20260926 03:06:53.477555 8508 ts_tablet_manager.cc:1397] T a3bfb8c9387e46e695a625bb34f2a02c P d8e10deebf684e65a687a362c3ff6f92: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.009s
I20260926 03:06:53.477844 8505 tablet_bootstrap.cc:492] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327: No bootstrap required, opened a new log
I20260926 03:06:53.478008 8505 ts_tablet_manager.cc:1397] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20260926 03:06:53.478096 8506 raft_consensus.cc:348] T a3bfb8c9387e46e695a625bb34f2a02c P 6ce2bd18989b4dfd89abcb754802e625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:53.478325 8506 raft_consensus.cc:374] T a3bfb8c9387e46e695a625bb34f2a02c P 6ce2bd18989b4dfd89abcb754802e625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:53.478418 8506 raft_consensus.cc:729] T a3bfb8c9387e46e695a625bb34f2a02c P 6ce2bd18989b4dfd89abcb754802e625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ce2bd18989b4dfd89abcb754802e625, State: Initialized, Role: FOLLOWER
I20260926 03:06:53.478634 8508 raft_consensus.cc:348] T a3bfb8c9387e46e695a625bb34f2a02c P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:53.478863 8508 raft_consensus.cc:374] T a3bfb8c9387e46e695a625bb34f2a02c P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:53.478760 8506 consensus_queue.cc:260] T a3bfb8c9387e46e695a625bb34f2a02c P 6ce2bd18989b4dfd89abcb754802e625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:53.479362 8506 ts_tablet_manager.cc:1428] T a3bfb8c9387e46e695a625bb34f2a02c P 6ce2bd18989b4dfd89abcb754802e625: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 03:06:53.479557 8505 raft_consensus.cc:348] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:53.479635 8508 raft_consensus.cc:729] T a3bfb8c9387e46e695a625bb34f2a02c P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8e10deebf684e65a687a362c3ff6f92, State: Initialized, Role: FOLLOWER
I20260926 03:06:53.479759 8505 raft_consensus.cc:374] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:53.479854 8505 raft_consensus.cc:729] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7a8e25854b0414e8fd86eb9fc11b327, State: Initialized, Role: FOLLOWER
I20260926 03:06:53.479897 8508 consensus_queue.cc:260] T a3bfb8c9387e46e695a625bb34f2a02c P d8e10deebf684e65a687a362c3ff6f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:53.480098 8505 consensus_queue.cc:260] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:53.480700 8505 ts_tablet_manager.cc:1428] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 03:06:53.481098 8508 ts_tablet_manager.cc:1428] T a3bfb8c9387e46e695a625bb34f2a02c P d8e10deebf684e65a687a362c3ff6f92: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 03:06:53.548511 8512 raft_consensus.cc:482] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:06:53.548777 8512 raft_consensus.cc:504] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:53.550684 8512 leader_election.cc:288] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ce2bd18989b4dfd89abcb754802e625 (127.0.97.1:32987), d8e10deebf684e65a687a362c3ff6f92 (127.0.97.2:44011)
I20260926 03:06:53.559407 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:53.551543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52409 (local address 127.0.97.2:44011)
0926 03:06:53.551710 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:53.551716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:53.552529 (+ 813us) server_negotiation.cc:407] Connection header received
0926 03:06:53.552683 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:53.552688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:53.552753 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:53.552846 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:53.553454 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.554525 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.556624 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.557994 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.558153 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:53.558245 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:53.559030 (+ 785us) server_negotiation.cc:299] Negotiation successful
0926 03:06:53.559151 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:06:53.559527 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:53.551108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44011 (local address 127.0.97.3:52409)
0926 03:06:53.552404 (+ 1296us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:53.552422 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:53.552553 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:53.552967 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:53.552970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:53.552989 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:53.553268 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:53.553278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.554687 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.554691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:53.556457 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:53.556475 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.558665 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.558671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:53.558858 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:53.558892 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 03:06:53.558989 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 03:06:53.559138 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1188,"mutex_wait_us":32,"thread_start_us":764,"threads_started":1}
I20260926 03:06:53.560243 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:53.551287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59677 (local address 127.0.97.1:32987)
0926 03:06:53.551458 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:53.551465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:53.552255 (+ 790us) server_negotiation.cc:407] Connection header received
0926 03:06:53.552513 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:53.552518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:53.552604 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:53.552692 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:53.553092 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.554196 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.556685 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.558020 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.558147 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:53.558250 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:53.559868 (+ 1618us) server_negotiation.cc:299] Negotiation successful
0926 03:06:53.560022 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 03:06:53.560475 8292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bfb8c9387e46e695a625bb34f2a02c" candidate_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8e10deebf684e65a687a362c3ff6f92" is_pre_election: true
I20260926 03:06:53.560989 8292 raft_consensus.cc:2454] T a3bfb8c9387e46e695a625bb34f2a02c P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7a8e25854b0414e8fd86eb9fc11b327 in term 0.
I20260926 03:06:53.561681 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:53.550872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32987 (local address 127.0.97.3:59677)
0926 03:06:53.552126 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:53.552149 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:53.552364 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:53.552724 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:53.552728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:53.552752 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:53.552943 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:53.552955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.554329 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.554342 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:53.556534 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:53.556551 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.558194 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.558201 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:53.558408 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:53.558446 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 03:06:53.559801 (+ 1355us) client_negotiation.cc:240] Negotiation successful
0926 03:06:53.559947 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1132,"mutex_wait_us":33,"thread_start_us":1273,"threads_started":1}
I20260926 03:06:53.561720 8365 leader_election.cc:302] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8e10deebf684e65a687a362c3ff6f92, f7a8e25854b0414e8fd86eb9fc11b327; no voters:
I20260926 03:06:53.562198 8512 raft_consensus.cc:2781] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:06:53.562318 8512 raft_consensus.cc:482] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:06:53.562417 8512 raft_consensus.cc:3037] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:53.562541 8158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bfb8c9387e46e695a625bb34f2a02c" candidate_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ce2bd18989b4dfd89abcb754802e625" is_pre_election: true
I20260926 03:06:53.562788 8158 raft_consensus.cc:2454] T a3bfb8c9387e46e695a625bb34f2a02c P 6ce2bd18989b4dfd89abcb754802e625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7a8e25854b0414e8fd86eb9fc11b327 in term 0.
I20260926 03:06:53.565647 8512 raft_consensus.cc:504] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:53.566529 8512 leader_election.cc:288] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327 [CANDIDATE]: Term 1 election: Requested vote from peers 6ce2bd18989b4dfd89abcb754802e625 (127.0.97.1:32987), d8e10deebf684e65a687a362c3ff6f92 (127.0.97.2:44011)
I20260926 03:06:53.567088 8158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bfb8c9387e46e695a625bb34f2a02c" candidate_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ce2bd18989b4dfd89abcb754802e625"
I20260926 03:06:53.567152 8292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bfb8c9387e46e695a625bb34f2a02c" candidate_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8e10deebf684e65a687a362c3ff6f92"
I20260926 03:06:53.567329 8158 raft_consensus.cc:3037] T a3bfb8c9387e46e695a625bb34f2a02c P 6ce2bd18989b4dfd89abcb754802e625 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:53.567401 8292 raft_consensus.cc:3037] T a3bfb8c9387e46e695a625bb34f2a02c P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:53.570066 8292 raft_consensus.cc:2454] T a3bfb8c9387e46e695a625bb34f2a02c P d8e10deebf684e65a687a362c3ff6f92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7a8e25854b0414e8fd86eb9fc11b327 in term 1.
I20260926 03:06:53.570307 8158 raft_consensus.cc:2454] T a3bfb8c9387e46e695a625bb34f2a02c P 6ce2bd18989b4dfd89abcb754802e625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7a8e25854b0414e8fd86eb9fc11b327 in term 1.
I20260926 03:06:53.570567 8365 leader_election.cc:302] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8e10deebf684e65a687a362c3ff6f92, f7a8e25854b0414e8fd86eb9fc11b327; no voters:
I20260926 03:06:53.570930 8512 raft_consensus.cc:2781] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:53.571367 8512 raft_consensus.cc:686] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327 [term 1 LEADER]: Becoming Leader. State: Replica: f7a8e25854b0414e8fd86eb9fc11b327, State: Running, Role: LEADER
I20260926 03:06:53.571787 8512 consensus_queue.cc:237] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:53.575762 7995 catalog_manager.cc:5495] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7a8e25854b0414e8fd86eb9fc11b327 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } health_report { overall_health: HEALTHY } } }
Sep 26 03:06:53 dist-test-slave-hmbw krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392011, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:06:53.637140 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:53.621399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46753 (local address 127.0.0.1:50876)
0926 03:06:53.622125 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:53.622142 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:53.622282 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:53.622659 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:53.622663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:53.623381 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:53.623646 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:53.623655 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.625185 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.625189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:53.626226 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:53.626239 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.627232 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.627239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:53.627375 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:53.628103 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:53.628126 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:53.630559 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:53.634279 (+ 3720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:53.634299 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:53.634308 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:53.634852 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:53.635339 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:53.635344 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:53.635348 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:53.635543 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:53.636174 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:53.636182 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:53.636358 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 03:06:53.636502 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 03:06:53.636744 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":406,"threads_started":1}
I20260926 03:06:53.637326 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:53.621236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50876 (local address 127.0.97.62:46753)
0926 03:06:53.621520 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:53.621524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:53.622247 (+ 723us) server_negotiation.cc:407] Connection header received
0926 03:06:53.622387 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:53.622392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:53.622471 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:53.622620 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:53.623801 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.625053 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.626357 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:53.627019 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:53.627148 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:53.630767 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:53.630774 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:53.630780 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:53.630819 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:53.634087 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:53.635022 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:53.635028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:53.635032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:53.635210 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:53.635656 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:53.635661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:53.635665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:53.636056 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:53.636202 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:53.636643 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 03:06:53.636870 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260926 03:06:53.643628 7995 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50876:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 03:06:53.644949 7995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:06:53.652827 8133 tablet_service.cc:1467] Processing CreateTablet for tablet 6580a5a828f34cb49c08f08614110a01 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9e3bc0704cd54e8ca39f1900a6d1f927]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:06:53.653008 8411 tablet_service.cc:1467] Processing CreateTablet for tablet 6580a5a828f34cb49c08f08614110a01 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9e3bc0704cd54e8ca39f1900a6d1f927]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:06:53.653373 8133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6580a5a828f34cb49c08f08614110a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:53.653523 8411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6580a5a828f34cb49c08f08614110a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:53.659317 8272 tablet_service.cc:1467] Processing CreateTablet for tablet 6580a5a828f34cb49c08f08614110a01 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9e3bc0704cd54e8ca39f1900a6d1f927]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:06:53.659951 8272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6580a5a828f34cb49c08f08614110a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:53.660779 8505 tablet_bootstrap.cc:492] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327: Bootstrap starting.
I20260926 03:06:53.662540 8505 tablet_bootstrap.cc:654] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:53.662750 8506 tablet_bootstrap.cc:492] T 6580a5a828f34cb49c08f08614110a01 P 6ce2bd18989b4dfd89abcb754802e625: Bootstrap starting.
I20260926 03:06:53.664386 8506 tablet_bootstrap.cc:654] T 6580a5a828f34cb49c08f08614110a01 P 6ce2bd18989b4dfd89abcb754802e625: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:53.664940 8505 tablet_bootstrap.cc:492] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327: No bootstrap required, opened a new log
I20260926 03:06:53.665095 8505 ts_tablet_manager.cc:1397] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260926 03:06:53.665840 8505 raft_consensus.cc:348] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:53.666091 8505 raft_consensus.cc:374] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:53.666167 8505 raft_consensus.cc:729] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7a8e25854b0414e8fd86eb9fc11b327, State: Initialized, Role: FOLLOWER
I20260926 03:06:53.666406 8505 consensus_queue.cc:260] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:53.666862 8512 tablet_replica.cc:439] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } } }
I20260926 03:06:53.667140 8512 tablet_replica.cc:442] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:06:53.668234 8505 ts_tablet_manager.cc:1428] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260926 03:06:53.669890 8506 tablet_bootstrap.cc:492] T 6580a5a828f34cb49c08f08614110a01 P 6ce2bd18989b4dfd89abcb754802e625: No bootstrap required, opened a new log
I20260926 03:06:53.670122 8506 ts_tablet_manager.cc:1397] T 6580a5a828f34cb49c08f08614110a01 P 6ce2bd18989b4dfd89abcb754802e625: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20260926 03:06:53.670710 8508 tablet_bootstrap.cc:492] T 6580a5a828f34cb49c08f08614110a01 P d8e10deebf684e65a687a362c3ff6f92: Bootstrap starting.
I20260926 03:06:53.670964 8506 raft_consensus.cc:348] T 6580a5a828f34cb49c08f08614110a01 P 6ce2bd18989b4dfd89abcb754802e625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:53.671415 8506 raft_consensus.cc:374] T 6580a5a828f34cb49c08f08614110a01 P 6ce2bd18989b4dfd89abcb754802e625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:53.671531 8506 raft_consensus.cc:729] T 6580a5a828f34cb49c08f08614110a01 P 6ce2bd18989b4dfd89abcb754802e625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ce2bd18989b4dfd89abcb754802e625, State: Initialized, Role: FOLLOWER
I20260926 03:06:53.671800 8506 consensus_queue.cc:260] T 6580a5a828f34cb49c08f08614110a01 P 6ce2bd18989b4dfd89abcb754802e625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:53.672322 8508 tablet_bootstrap.cc:654] T 6580a5a828f34cb49c08f08614110a01 P d8e10deebf684e65a687a362c3ff6f92: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:53.672322 8506 ts_tablet_manager.cc:1428] T 6580a5a828f34cb49c08f08614110a01 P 6ce2bd18989b4dfd89abcb754802e625: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 03:06:53.673581 8511 tablet_replica.cc:439] T 6580a5a828f34cb49c08f08614110a01 P 6ce2bd18989b4dfd89abcb754802e625: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } } }
I20260926 03:06:53.673930 8511 tablet_replica.cc:442] T 6580a5a828f34cb49c08f08614110a01 P 6ce2bd18989b4dfd89abcb754802e625: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:06:53.676036 8508 tablet_bootstrap.cc:492] T 6580a5a828f34cb49c08f08614110a01 P d8e10deebf684e65a687a362c3ff6f92: No bootstrap required, opened a new log
I20260926 03:06:53.676209 8508 ts_tablet_manager.cc:1397] T 6580a5a828f34cb49c08f08614110a01 P d8e10deebf684e65a687a362c3ff6f92: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 03:06:53.677011 8508 raft_consensus.cc:348] T 6580a5a828f34cb49c08f08614110a01 P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:53.677253 8508 raft_consensus.cc:374] T 6580a5a828f34cb49c08f08614110a01 P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:53.677333 8508 raft_consensus.cc:729] T 6580a5a828f34cb49c08f08614110a01 P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8e10deebf684e65a687a362c3ff6f92, State: Initialized, Role: FOLLOWER
I20260926 03:06:53.677621 8508 consensus_queue.cc:260] T 6580a5a828f34cb49c08f08614110a01 P d8e10deebf684e65a687a362c3ff6f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:53.678261 8508 ts_tablet_manager.cc:1428] T 6580a5a828f34cb49c08f08614110a01 P d8e10deebf684e65a687a362c3ff6f92: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 03:06:53.678071 8516 tablet_replica.cc:439] T 6580a5a828f34cb49c08f08614110a01 P d8e10deebf684e65a687a362c3ff6f92: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } } }
I20260926 03:06:53.678433 8516 tablet_replica.cc:442] T 6580a5a828f34cb49c08f08614110a01 P d8e10deebf684e65a687a362c3ff6f92: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:06:54.020035 8512 raft_consensus.cc:482] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:06:54.020273 8512 raft_consensus.cc:504] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:54.020924 8512 leader_election.cc:288] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ce2bd18989b4dfd89abcb754802e625 (127.0.97.1:32987), d8e10deebf684e65a687a362c3ff6f92 (127.0.97.2:44011)
I20260926 03:06:54.021538 8158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6580a5a828f34cb49c08f08614110a01" candidate_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ce2bd18989b4dfd89abcb754802e625" is_pre_election: true
I20260926 03:06:54.021533 8292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6580a5a828f34cb49c08f08614110a01" candidate_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8e10deebf684e65a687a362c3ff6f92" is_pre_election: true
I20260926 03:06:54.021785 8292 raft_consensus.cc:2454] T 6580a5a828f34cb49c08f08614110a01 P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7a8e25854b0414e8fd86eb9fc11b327 in term 0.
I20260926 03:06:54.021791 8158 raft_consensus.cc:2454] T 6580a5a828f34cb49c08f08614110a01 P 6ce2bd18989b4dfd89abcb754802e625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7a8e25854b0414e8fd86eb9fc11b327 in term 0.
I20260926 03:06:54.022249 8365 leader_election.cc:302] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8e10deebf684e65a687a362c3ff6f92, f7a8e25854b0414e8fd86eb9fc11b327; no voters:
I20260926 03:06:54.022596 8512 raft_consensus.cc:2781] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:06:54.022696 8512 raft_consensus.cc:482] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:06:54.022789 8512 raft_consensus.cc:3037] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:54.025597 8512 raft_consensus.cc:504] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:54.026315 8512 leader_election.cc:288] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327 [CANDIDATE]: Term 1 election: Requested vote from peers 6ce2bd18989b4dfd89abcb754802e625 (127.0.97.1:32987), d8e10deebf684e65a687a362c3ff6f92 (127.0.97.2:44011)
I20260926 03:06:54.026842 8292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6580a5a828f34cb49c08f08614110a01" candidate_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8e10deebf684e65a687a362c3ff6f92"
I20260926 03:06:54.027086 8292 raft_consensus.cc:3037] T 6580a5a828f34cb49c08f08614110a01 P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:54.028757 8158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6580a5a828f34cb49c08f08614110a01" candidate_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ce2bd18989b4dfd89abcb754802e625"
I20260926 03:06:54.029043 8158 raft_consensus.cc:3037] T 6580a5a828f34cb49c08f08614110a01 P 6ce2bd18989b4dfd89abcb754802e625 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:54.030196 8292 raft_consensus.cc:2454] T 6580a5a828f34cb49c08f08614110a01 P d8e10deebf684e65a687a362c3ff6f92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7a8e25854b0414e8fd86eb9fc11b327 in term 1.
I20260926 03:06:54.030757 8365 leader_election.cc:302] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8e10deebf684e65a687a362c3ff6f92, f7a8e25854b0414e8fd86eb9fc11b327; no voters:
I20260926 03:06:54.031062 8512 raft_consensus.cc:2781] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:54.031215 8512 raft_consensus.cc:686] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327 [term 1 LEADER]: Becoming Leader. State: Replica: f7a8e25854b0414e8fd86eb9fc11b327, State: Running, Role: LEADER
I20260926 03:06:54.031535 8158 raft_consensus.cc:2454] T 6580a5a828f34cb49c08f08614110a01 P 6ce2bd18989b4dfd89abcb754802e625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7a8e25854b0414e8fd86eb9fc11b327 in term 1.
I20260926 03:06:54.031481 8512 consensus_queue.cc:237] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:54.033113 8536 tablet_replica.cc:439] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327: TxnStatusTablet state changed. Reason: New leader f7a8e25854b0414e8fd86eb9fc11b327. Latest consensus state: current_term: 1 leader_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } } }
I20260926 03:06:54.033404 8536 tablet_replica.cc:442] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:06:54.034031 8548 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 03:06:54.034263 8548 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 03:06:54.034942 7994 catalog_manager.cc:5495] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7a8e25854b0414e8fd86eb9fc11b327 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } health_report { overall_health: HEALTHY } } }
I20260926 03:06:54.077023 8536 consensus_queue.cc:1035] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:54.084442 8553 consensus_queue.cc:1035] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:54.133415 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:54.126704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57578 (local address 127.0.97.3:34895)
0926 03:06:54.127290 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:54.127294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:54.127309 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:06:54.127388 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:54.127391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:54.127474 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:54.127603 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:06:54.128104 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.128994 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.130932 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.131399 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.131496 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:54.132035 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:06:54.132799 (+ 764us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:06:54.132932 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:54.133068 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 03:06:54.133137 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":333,"threads_started":1}
I20260926 03:06:54.133503 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:54.126616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34895 (local address 127.0.0.1:57578)
0926 03:06:54.126875 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:54.126896 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:54.127057 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:54.127647 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:54.127652 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:54.127666 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:06:54.127868 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:54.127879 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.129134 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.129140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:54.130786 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:54.130802 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.131560 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.131566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:54.131759 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:54.131843 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:06:54.132908 (+ 1065us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:06:54.132930 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 03:06:54.133016 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 03:06:54.133179 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 03:06:54.137497 8158 raft_consensus.cc:1261] T 6580a5a828f34cb49c08f08614110a01 P 6ce2bd18989b4dfd89abcb754802e625 [term 1 FOLLOWER]: Refusing update from remote peer f7a8e25854b0414e8fd86eb9fc11b327: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:54.137476 8292 raft_consensus.cc:1261] T 6580a5a828f34cb49c08f08614110a01 P d8e10deebf684e65a687a362c3ff6f92 [term 1 FOLLOWER]: Refusing update from remote peer f7a8e25854b0414e8fd86eb9fc11b327: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:54.138082 8553 consensus_queue.cc:1035] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:54.138451 8536 consensus_queue.cc:1035] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:54.141149 8511 tablet_replica.cc:439] T 6580a5a828f34cb49c08f08614110a01 P 6ce2bd18989b4dfd89abcb754802e625: TxnStatusTablet state changed. Reason: New leader f7a8e25854b0414e8fd86eb9fc11b327. Latest consensus state: current_term: 1 leader_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } } }
I20260926 03:06:54.141424 8511 tablet_replica.cc:442] T 6580a5a828f34cb49c08f08614110a01 P 6ce2bd18989b4dfd89abcb754802e625: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:06:54.148689 8553 tablet_replica.cc:439] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } } }
I20260926 03:06:54.149114 8553 tablet_replica.cc:442] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:06:54.154066 8536 tablet_replica.cc:439] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } } }
I20260926 03:06:54.154341 8536 tablet_replica.cc:442] T 6580a5a828f34cb49c08f08614110a01 P f7a8e25854b0414e8fd86eb9fc11b327: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:06:54.159536 8511 tablet_replica.cc:439] T 6580a5a828f34cb49c08f08614110a01 P 6ce2bd18989b4dfd89abcb754802e625: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } } }
I20260926 03:06:54.159816 8511 tablet_replica.cc:442] T 6580a5a828f34cb49c08f08614110a01 P 6ce2bd18989b4dfd89abcb754802e625: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:06:54.163400 8516 tablet_replica.cc:439] T 6580a5a828f34cb49c08f08614110a01 P d8e10deebf684e65a687a362c3ff6f92: TxnStatusTablet state changed. Reason: New leader f7a8e25854b0414e8fd86eb9fc11b327. Latest consensus state: current_term: 1 leader_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } } }
I20260926 03:06:54.163687 8516 tablet_replica.cc:442] T 6580a5a828f34cb49c08f08614110a01 P d8e10deebf684e65a687a362c3ff6f92: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:06:54.164069 8562 tablet_replica.cc:439] T 6580a5a828f34cb49c08f08614110a01 P d8e10deebf684e65a687a362c3ff6f92: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } } }
I20260926 03:06:54.164335 8562 tablet_replica.cc:442] T 6580a5a828f34cb49c08f08614110a01 P d8e10deebf684e65a687a362c3ff6f92: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:06:54.172739 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:54.167750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57594 (local address 127.0.97.3:34895)
0926 03:06:54.167919 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:54.167925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:54.168073 (+ 148us) server_negotiation.cc:407] Connection header received
0926 03:06:54.168295 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:54.168301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:54.168404 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:54.168499 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:06:54.168949 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.169737 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.171077 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.171506 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.171617 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:54.172016 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:06:54.172136 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:06:54.172216 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:54.172344 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 03:06:54.172429 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:06:54.189729 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:54.183542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44863 (local address 127.0.97.3:34895)
0926 03:06:54.183717 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:54.183723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:54.184310 (+ 587us) server_negotiation.cc:407] Connection header received
0926 03:06:54.184510 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:54.184514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:54.184586 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:54.184662 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:06:54.185137 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.185855 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.188034 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.188448 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.188549 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:54.189079 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:06:54.189178 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:06:54.189247 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:54.189375 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 03:06:54.189438 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 03:06:54.190018 8568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:54.183532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34895 (local address 127.0.97.3:44863)
0926 03:06:54.184204 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:54.184223 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:54.184381 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:54.184694 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:54.184698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:54.184711 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:06:54.184997 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:54.185010 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.186031 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.186037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:54.187895 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:54.187910 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.188626 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.188633 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:54.188802 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:54.188880 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:06:54.189272 (+ 392us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:06:54.189284 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 03:06:54.189369 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 03:06:54.189537 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":1,"thread_start_us":352,"threads_started":1}
I20260926 03:06:54.255707 7995 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:06:54.255988 7995 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:06:54.259200 7995 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 665ebc9724d34582a9f1ba3490f657ab: Sending DeleteTablet for 3 replicas of tablet a3bfb8c9387e46e695a625bb34f2a02c
I20260926 03:06:54.260020 8133 tablet_service.cc:1514] Processing DeleteTablet for tablet a3bfb8c9387e46e695a625bb34f2a02c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:54 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49838
I20260926 03:06:54.260298 8411 tablet_service.cc:1514] Processing DeleteTablet for tablet a3bfb8c9387e46e695a625bb34f2a02c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:54 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57562
I20260926 03:06:54.260421 8272 tablet_service.cc:1514] Processing DeleteTablet for tablet a3bfb8c9387e46e695a625bb34f2a02c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:54 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51228
I20260926 03:06:54.260777 8570 tablet_replica.cc:330] T a3bfb8c9387e46e695a625bb34f2a02c P 6ce2bd18989b4dfd89abcb754802e625: stopping tablet replica
I20260926 03:06:54.269022 8570 raft_consensus.cc:2229] T a3bfb8c9387e46e695a625bb34f2a02c P 6ce2bd18989b4dfd89abcb754802e625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:06:54.269532 8570 raft_consensus.cc:2258] T a3bfb8c9387e46e695a625bb34f2a02c P 6ce2bd18989b4dfd89abcb754802e625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:54.270912 8572 tablet_replica.cc:330] T a3bfb8c9387e46e695a625bb34f2a02c P d8e10deebf684e65a687a362c3ff6f92: stopping tablet replica
I20260926 03:06:54.271116 8572 raft_consensus.cc:2229] T a3bfb8c9387e46e695a625bb34f2a02c P d8e10deebf684e65a687a362c3ff6f92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:06:54.271311 8572 raft_consensus.cc:2258] T a3bfb8c9387e46e695a625bb34f2a02c P d8e10deebf684e65a687a362c3ff6f92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:54.271992 8571 tablet_replica.cc:330] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327: stopping tablet replica
I20260926 03:06:54.272611 8571 raft_consensus.cc:2229] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:06:54.273041 8572 ts_tablet_manager.cc:1905] T a3bfb8c9387e46e695a625bb34f2a02c P d8e10deebf684e65a687a362c3ff6f92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:54.273185 8571 raft_consensus.cc:2258] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:54.274250 8571 ts_tablet_manager.cc:1905] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:54.274273 8570 ts_tablet_manager.cc:1905] T a3bfb8c9387e46e695a625bb34f2a02c P 6ce2bd18989b4dfd89abcb754802e625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:54.280222 8572 ts_tablet_manager.cc:1918] T a3bfb8c9387e46e695a625bb34f2a02c P d8e10deebf684e65a687a362c3ff6f92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:06:54.280449 8572 log.cc:1196] T a3bfb8c9387e46e695a625bb34f2a02c P d8e10deebf684e65a687a362c3ff6f92: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-1/wal/wals/a3bfb8c9387e46e695a625bb34f2a02c
I20260926 03:06:54.280849 8570 ts_tablet_manager.cc:1918] T a3bfb8c9387e46e695a625bb34f2a02c P 6ce2bd18989b4dfd89abcb754802e625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:06:54.280997 8570 log.cc:1196] T a3bfb8c9387e46e695a625bb34f2a02c P 6ce2bd18989b4dfd89abcb754802e625: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-0/wal/wals/a3bfb8c9387e46e695a625bb34f2a02c
I20260926 03:06:54.281163 8571 ts_tablet_manager.cc:1918] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:06:54.281200 8572 ts_tablet_manager.cc:1939] T a3bfb8c9387e46e695a625bb34f2a02c P d8e10deebf684e65a687a362c3ff6f92: Deleting consensus metadata
I20260926 03:06:54.281306 8571 log.cc:1196] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-2/wal/wals/a3bfb8c9387e46e695a625bb34f2a02c
I20260926 03:06:54.281409 8570 ts_tablet_manager.cc:1939] T a3bfb8c9387e46e695a625bb34f2a02c P 6ce2bd18989b4dfd89abcb754802e625: Deleting consensus metadata
I20260926 03:06:54.281735 8571 ts_tablet_manager.cc:1939] T a3bfb8c9387e46e695a625bb34f2a02c P f7a8e25854b0414e8fd86eb9fc11b327: Deleting consensus metadata
I20260926 03:06:54.282545 7981 catalog_manager.cc:4841] TS 6ce2bd18989b4dfd89abcb754802e625 (127.0.97.1:32987): tablet a3bfb8c9387e46e695a625bb34f2a02c (table test-table [id=322e1349f211402fade0a5a441086aeb]) successfully deleted
I20260926 03:06:54.282568 7983 catalog_manager.cc:4841] TS d8e10deebf684e65a687a362c3ff6f92 (127.0.97.2:44011): tablet a3bfb8c9387e46e695a625bb34f2a02c (table test-table [id=322e1349f211402fade0a5a441086aeb]) successfully deleted
I20260926 03:06:54.283062 7983 catalog_manager.cc:4841] TS f7a8e25854b0414e8fd86eb9fc11b327 (127.0.97.3:34895): tablet a3bfb8c9387e46e695a625bb34f2a02c (table test-table [id=322e1349f211402fade0a5a441086aeb]) successfully deleted
Sep 26 03:06:54 dist-test-slave-hmbw krb5kdc[7951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 03:06:54 dist-test-slave-hmbw krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:06:54 dist-test-slave-hmbw krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:06:54.332870 8580 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:06:54.321324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50878 (local address 127.0.97.62:46753)
0926 03:06:54.322035 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:54.322040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:54.322057 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:06:54.322108 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:54.322112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:54.322188 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:54.322291 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:54.323953 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.324822 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.326701 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.327141 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.327245 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:54.332548 (+ 5303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50878: BlockingRecv error: recv got EOF from 127.0.0.1:50878 (error 108)
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":460,"threads_started":1}
Sep 26 03:06:54 dist-test-slave-hmbw krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:06:54.349992 8580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:54.335901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50890 (local address 127.0.97.62:46753)
0926 03:06:54.336058 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:54.336063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:54.336394 (+ 331us) server_negotiation.cc:407] Connection header received
0926 03:06:54.336571 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:54.336577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:54.336666 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:54.336764 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:54.338156 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.338974 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.340219 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.340705 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.340819 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:54.344713 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:54.344719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:54.344722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:54.344759 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:54.347474 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:54.348123 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:54.348129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:54.348131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:54.348258 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:54.348659 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:54.348663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:54.348665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:54.348946 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:54.349168 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:54.349465 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 03:06:54.349672 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 03:06:54.353654 7995 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:06:54.354223 7995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:06:54.361169 8133 tablet_service.cc:1467] Processing CreateTablet for tablet c52b5de6329a446eaab7dc4bfe4e8cfe (DEFAULT_TABLE table=test-table [id=243b1d05d7b84862afc95707b3cc3ee0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:54.361639 8133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c52b5de6329a446eaab7dc4bfe4e8cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:54.361580 8272 tablet_service.cc:1467] Processing CreateTablet for tablet c52b5de6329a446eaab7dc4bfe4e8cfe (DEFAULT_TABLE table=test-table [id=243b1d05d7b84862afc95707b3cc3ee0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:54.361948 8272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c52b5de6329a446eaab7dc4bfe4e8cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:54.362067 8411 tablet_service.cc:1467] Processing CreateTablet for tablet c52b5de6329a446eaab7dc4bfe4e8cfe (DEFAULT_TABLE table=test-table [id=243b1d05d7b84862afc95707b3cc3ee0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:54.362478 8411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c52b5de6329a446eaab7dc4bfe4e8cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:54.373679 8586 tablet_bootstrap.cc:492] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625: Bootstrap starting.
I20260926 03:06:54.374034 8588 tablet_bootstrap.cc:492] T c52b5de6329a446eaab7dc4bfe4e8cfe P f7a8e25854b0414e8fd86eb9fc11b327: Bootstrap starting.
I20260926 03:06:54.375722 8586 tablet_bootstrap.cc:654] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:54.376044 8587 tablet_bootstrap.cc:492] T c52b5de6329a446eaab7dc4bfe4e8cfe P d8e10deebf684e65a687a362c3ff6f92: Bootstrap starting.
I20260926 03:06:54.377552 8587 tablet_bootstrap.cc:654] T c52b5de6329a446eaab7dc4bfe4e8cfe P d8e10deebf684e65a687a362c3ff6f92: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:54.378060 8586 tablet_bootstrap.cc:492] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625: No bootstrap required, opened a new log
I20260926 03:06:54.378203 8586 ts_tablet_manager.cc:1397] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 03:06:54.379035 8586 raft_consensus.cc:348] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:54.379289 8586 raft_consensus.cc:374] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:54.379377 8586 raft_consensus.cc:729] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ce2bd18989b4dfd89abcb754802e625, State: Initialized, Role: FOLLOWER
I20260926 03:06:54.379614 8586 consensus_queue.cc:260] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:54.380164 8588 tablet_bootstrap.cc:654] T c52b5de6329a446eaab7dc4bfe4e8cfe P f7a8e25854b0414e8fd86eb9fc11b327: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:54.380815 8586 ts_tablet_manager.cc:1428] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 03:06:54.381807 8587 tablet_bootstrap.cc:492] T c52b5de6329a446eaab7dc4bfe4e8cfe P d8e10deebf684e65a687a362c3ff6f92: No bootstrap required, opened a new log
I20260926 03:06:54.381969 8587 ts_tablet_manager.cc:1397] T c52b5de6329a446eaab7dc4bfe4e8cfe P d8e10deebf684e65a687a362c3ff6f92: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260926 03:06:54.382710 8587 raft_consensus.cc:348] T c52b5de6329a446eaab7dc4bfe4e8cfe P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:54.382988 8587 raft_consensus.cc:374] T c52b5de6329a446eaab7dc4bfe4e8cfe P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:54.383083 8587 raft_consensus.cc:729] T c52b5de6329a446eaab7dc4bfe4e8cfe P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8e10deebf684e65a687a362c3ff6f92, State: Initialized, Role: FOLLOWER
I20260926 03:06:54.383311 8587 consensus_queue.cc:260] T c52b5de6329a446eaab7dc4bfe4e8cfe P d8e10deebf684e65a687a362c3ff6f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:54.383980 8587 ts_tablet_manager.cc:1428] T c52b5de6329a446eaab7dc4bfe4e8cfe P d8e10deebf684e65a687a362c3ff6f92: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:06:54.388320 8588 tablet_bootstrap.cc:492] T c52b5de6329a446eaab7dc4bfe4e8cfe P f7a8e25854b0414e8fd86eb9fc11b327: No bootstrap required, opened a new log
I20260926 03:06:54.388484 8588 ts_tablet_manager.cc:1397] T c52b5de6329a446eaab7dc4bfe4e8cfe P f7a8e25854b0414e8fd86eb9fc11b327: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20260926 03:06:54.389189 8588 raft_consensus.cc:348] T c52b5de6329a446eaab7dc4bfe4e8cfe P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:54.389402 8588 raft_consensus.cc:374] T c52b5de6329a446eaab7dc4bfe4e8cfe P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:54.389500 8588 raft_consensus.cc:729] T c52b5de6329a446eaab7dc4bfe4e8cfe P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7a8e25854b0414e8fd86eb9fc11b327, State: Initialized, Role: FOLLOWER
I20260926 03:06:54.389758 8588 consensus_queue.cc:260] T c52b5de6329a446eaab7dc4bfe4e8cfe P f7a8e25854b0414e8fd86eb9fc11b327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:54.390296 8588 ts_tablet_manager.cc:1428] T c52b5de6329a446eaab7dc4bfe4e8cfe P f7a8e25854b0414e8fd86eb9fc11b327: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 03:06:54.437040 8511 raft_consensus.cc:482] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:06:54.437290 8511 raft_consensus.cc:504] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:54.439030 8511 leader_election.cc:288] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8e10deebf684e65a687a362c3ff6f92 (127.0.97.2:44011), f7a8e25854b0414e8fd86eb9fc11b327 (127.0.97.3:34895)
I20260926 03:06:54.447208 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:54.439369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47315 (local address 127.0.97.2:44011)
0926 03:06:54.440960 (+ 1591us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:54.440967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:54.440993 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:06:54.441070 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:54.441076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:54.441160 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:54.441267 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:54.441945 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.443029 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.444941 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.446156 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.446308 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:54.446436 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:54.446733 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 03:06:54.446884 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1458,"thread_start_us":1315,"threads_started":1}
I20260926 03:06:54.447250 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:54.439299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44011 (local address 127.0.97.1:47315)
0926 03:06:54.439911 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:54.439930 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:54.440056 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:54.441312 (+ 1256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:54.441317 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:54.441350 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:54.441751 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:54.441764 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.443145 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.443151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:54.444770 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:54.444785 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.446355 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.446362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:54.446562 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:54.446599 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 03:06:54.446694 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 03:06:54.446837 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":1,"thread_start_us":324,"threads_started":1}
I20260926 03:06:54.447953 8292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52b5de6329a446eaab7dc4bfe4e8cfe" candidate_uuid: "6ce2bd18989b4dfd89abcb754802e625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8e10deebf684e65a687a362c3ff6f92" is_pre_election: true
I20260926 03:06:54.448216 8292 raft_consensus.cc:2454] T c52b5de6329a446eaab7dc4bfe4e8cfe P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ce2bd18989b4dfd89abcb754802e625 in term 0.
I20260926 03:06:54.448635 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:54.440034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34895 (local address 127.0.97.1:42199)
0926 03:06:54.441881 (+ 1847us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:54.441900 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:54.442022 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:54.442348 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:54.442353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:54.442379 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:06:54.442728 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:54.442739 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.444078 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.444083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:54.445758 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:54.445770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.447826 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.447832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:54.448010 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:54.448034 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 03:06:54.448114 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 03:06:54.448241 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1736,"mutex_wait_us":46,"thread_start_us":1541,"threads_started":1}
I20260926 03:06:54.448879 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:54.440500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42199 (local address 127.0.97.3:34895)
0926 03:06:54.440716 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:54.440722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:54.441969 (+ 1247us) server_negotiation.cc:407] Connection header received
0926 03:06:54.442136 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:54.442142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:54.442216 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:54.442308 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:06:54.442886 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.443934 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.445908 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.447630 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.447766 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:54.447866 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:54.448248 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 03:06:54.448478 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:06:54.448791 8086 leader_election.cc:302] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ce2bd18989b4dfd89abcb754802e625, d8e10deebf684e65a687a362c3ff6f92; no voters:
I20260926 03:06:54.449184 8511 raft_consensus.cc:2781] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:06:54.449296 8511 raft_consensus.cc:482] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:06:54.449396 8511 raft_consensus.cc:3037] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:54.449753 8431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52b5de6329a446eaab7dc4bfe4e8cfe" candidate_uuid: "6ce2bd18989b4dfd89abcb754802e625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" is_pre_election: true
I20260926 03:06:54.450016 8431 raft_consensus.cc:2454] T c52b5de6329a446eaab7dc4bfe4e8cfe P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ce2bd18989b4dfd89abcb754802e625 in term 0.
I20260926 03:06:54.452597 8511 raft_consensus.cc:504] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:54.453181 8511 leader_election.cc:288] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625 [CANDIDATE]: Term 1 election: Requested vote from peers d8e10deebf684e65a687a362c3ff6f92 (127.0.97.2:44011), f7a8e25854b0414e8fd86eb9fc11b327 (127.0.97.3:34895)
I20260926 03:06:54.453771 8292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52b5de6329a446eaab7dc4bfe4e8cfe" candidate_uuid: "6ce2bd18989b4dfd89abcb754802e625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8e10deebf684e65a687a362c3ff6f92"
I20260926 03:06:54.453848 8431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52b5de6329a446eaab7dc4bfe4e8cfe" candidate_uuid: "6ce2bd18989b4dfd89abcb754802e625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7a8e25854b0414e8fd86eb9fc11b327"
I20260926 03:06:54.454044 8431 raft_consensus.cc:3037] T c52b5de6329a446eaab7dc4bfe4e8cfe P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:54.454036 8292 raft_consensus.cc:3037] T c52b5de6329a446eaab7dc4bfe4e8cfe P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:54.456161 8431 raft_consensus.cc:2454] T c52b5de6329a446eaab7dc4bfe4e8cfe P f7a8e25854b0414e8fd86eb9fc11b327 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ce2bd18989b4dfd89abcb754802e625 in term 1.
I20260926 03:06:54.456576 8086 leader_election.cc:302] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ce2bd18989b4dfd89abcb754802e625, f7a8e25854b0414e8fd86eb9fc11b327; no voters:
I20260926 03:06:54.456868 8511 raft_consensus.cc:2781] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:54.456887 8292 raft_consensus.cc:2454] T c52b5de6329a446eaab7dc4bfe4e8cfe P d8e10deebf684e65a687a362c3ff6f92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ce2bd18989b4dfd89abcb754802e625 in term 1.
I20260926 03:06:54.457367 8511 raft_consensus.cc:686] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625 [term 1 LEADER]: Becoming Leader. State: Replica: 6ce2bd18989b4dfd89abcb754802e625, State: Running, Role: LEADER
I20260926 03:06:54.457752 8511 consensus_queue.cc:237] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:54.461203 7995 catalog_manager.cc:5495] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ce2bd18989b4dfd89abcb754802e625 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "6ce2bd18989b4dfd89abcb754802e625" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } health_report { overall_health: UNKNOWN } } }
I20260926 03:06:54.483721 8598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:54.477316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45960 (local address 127.0.97.1:32987)
0926 03:06:54.477942 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:54.477947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:54.477970 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:06:54.478077 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:54.478082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:54.478178 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:54.478354 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:06:54.478763 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.479787 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.481102 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.481744 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.481865 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:54.482288 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:06:54.483054 (+ 766us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:06:54.483197 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:54.483288 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 03:06:54.483372 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":353,"threads_started":1}
I20260926 03:06:54.487721 8292 raft_consensus.cc:1261] T c52b5de6329a446eaab7dc4bfe4e8cfe P d8e10deebf684e65a687a362c3ff6f92 [term 1 FOLLOWER]: Refusing update from remote peer 6ce2bd18989b4dfd89abcb754802e625: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:54.487709 8431 raft_consensus.cc:1261] T c52b5de6329a446eaab7dc4bfe4e8cfe P f7a8e25854b0414e8fd86eb9fc11b327 [term 1 FOLLOWER]: Refusing update from remote peer 6ce2bd18989b4dfd89abcb754802e625: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:54.488451 8595 consensus_queue.cc:1035] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:54.488927 8511 consensus_queue.cc:1035] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:54.507961 7995 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:06:54.508213 7995 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:06:54.511168 7995 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 665ebc9724d34582a9f1ba3490f657ab: Sending DeleteTablet for 3 replicas of tablet c52b5de6329a446eaab7dc4bfe4e8cfe
I20260926 03:06:54.511986 8133 tablet_service.cc:1514] Processing DeleteTablet for tablet c52b5de6329a446eaab7dc4bfe4e8cfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:54 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49838
I20260926 03:06:54.512310 8570 tablet_replica.cc:330] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625: stopping tablet replica
I20260926 03:06:54.512485 8272 tablet_service.cc:1514] Processing DeleteTablet for tablet c52b5de6329a446eaab7dc4bfe4e8cfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:54 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51228
I20260926 03:06:54.512524 8570 raft_consensus.cc:2229] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:06:54.512763 8572 tablet_replica.cc:330] T c52b5de6329a446eaab7dc4bfe4e8cfe P d8e10deebf684e65a687a362c3ff6f92: stopping tablet replica
I20260926 03:06:54.512722 8411 tablet_service.cc:1514] Processing DeleteTablet for tablet c52b5de6329a446eaab7dc4bfe4e8cfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:54 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57562
I20260926 03:06:54.512949 8572 raft_consensus.cc:2229] T c52b5de6329a446eaab7dc4bfe4e8cfe P d8e10deebf684e65a687a362c3ff6f92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:06:54.512981 8571 tablet_replica.cc:330] T c52b5de6329a446eaab7dc4bfe4e8cfe P f7a8e25854b0414e8fd86eb9fc11b327: stopping tablet replica
I20260926 03:06:54.513077 8570 raft_consensus.cc:2258] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:54.513170 8572 raft_consensus.cc:2258] T c52b5de6329a446eaab7dc4bfe4e8cfe P d8e10deebf684e65a687a362c3ff6f92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:54.513160 8571 raft_consensus.cc:2229] T c52b5de6329a446eaab7dc4bfe4e8cfe P f7a8e25854b0414e8fd86eb9fc11b327 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:06:54.513392 8571 raft_consensus.cc:2258] T c52b5de6329a446eaab7dc4bfe4e8cfe P f7a8e25854b0414e8fd86eb9fc11b327 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:54.514689 7995 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 03:06:54.515065 8570 ts_tablet_manager.cc:1905] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 03:06:54.515599 7995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:06:54.516326 8572 ts_tablet_manager.cc:1905] T c52b5de6329a446eaab7dc4bfe4e8cfe P d8e10deebf684e65a687a362c3ff6f92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:54.523175 8572 ts_tablet_manager.cc:1918] T c52b5de6329a446eaab7dc4bfe4e8cfe P d8e10deebf684e65a687a362c3ff6f92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:06:54.523348 8572 log.cc:1196] T c52b5de6329a446eaab7dc4bfe4e8cfe P d8e10deebf684e65a687a362c3ff6f92: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-1/wal/wals/c52b5de6329a446eaab7dc4bfe4e8cfe
I20260926 03:06:54.523759 8572 ts_tablet_manager.cc:1939] T c52b5de6329a446eaab7dc4bfe4e8cfe P d8e10deebf684e65a687a362c3ff6f92: Deleting consensus metadata
I20260926 03:06:54.523756 8133 tablet_service.cc:1467] Processing CreateTablet for tablet e1b205ad566f4738958ce9ba171a9a1a (DEFAULT_TABLE table=test-table [id=c9061620f39f45299652fc8a76946f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:54.524196 8133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1b205ad566f4738958ce9ba171a9a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:54.524747 8411 tablet_service.cc:1467] Processing CreateTablet for tablet e1b205ad566f4738958ce9ba171a9a1a (DEFAULT_TABLE table=test-table [id=c9061620f39f45299652fc8a76946f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:54.525224 8411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1b205ad566f4738958ce9ba171a9a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:54.526747 8272 tablet_service.cc:1467] Processing CreateTablet for tablet e1b205ad566f4738958ce9ba171a9a1a (DEFAULT_TABLE table=test-table [id=c9061620f39f45299652fc8a76946f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:06:54.527184 8272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1b205ad566f4738958ce9ba171a9a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:54.527500 7983 catalog_manager.cc:4841] TS d8e10deebf684e65a687a362c3ff6f92 (127.0.97.2:44011): tablet c52b5de6329a446eaab7dc4bfe4e8cfe (table test-table [id=243b1d05d7b84862afc95707b3cc3ee0]) successfully deleted
I20260926 03:06:54.533638 8588 tablet_bootstrap.cc:492] T e1b205ad566f4738958ce9ba171a9a1a P f7a8e25854b0414e8fd86eb9fc11b327: Bootstrap starting.
I20260926 03:06:54.535398 8588 tablet_bootstrap.cc:654] T e1b205ad566f4738958ce9ba171a9a1a P f7a8e25854b0414e8fd86eb9fc11b327: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:54.536676 8587 tablet_bootstrap.cc:492] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92: Bootstrap starting.
I20260926 03:06:54.538374 8587 tablet_bootstrap.cc:654] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:54.539072 8586 tablet_bootstrap.cc:492] T e1b205ad566f4738958ce9ba171a9a1a P 6ce2bd18989b4dfd89abcb754802e625: Bootstrap starting.
I20260926 03:06:54.539582 8570 ts_tablet_manager.cc:1918] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:06:54.539728 8570 log.cc:1196] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-0/wal/wals/c52b5de6329a446eaab7dc4bfe4e8cfe
I20260926 03:06:54.540175 8570 ts_tablet_manager.cc:1939] T c52b5de6329a446eaab7dc4bfe4e8cfe P 6ce2bd18989b4dfd89abcb754802e625: Deleting consensus metadata
I20260926 03:06:54.540817 8586 tablet_bootstrap.cc:654] T e1b205ad566f4738958ce9ba171a9a1a P 6ce2bd18989b4dfd89abcb754802e625: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:54.542652 7981 catalog_manager.cc:4841] TS 6ce2bd18989b4dfd89abcb754802e625 (127.0.97.1:32987): tablet c52b5de6329a446eaab7dc4bfe4e8cfe (table test-table [id=243b1d05d7b84862afc95707b3cc3ee0]) successfully deleted
I20260926 03:06:54.543645 8587 tablet_bootstrap.cc:492] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92: No bootstrap required, opened a new log
I20260926 03:06:54.543905 8587 ts_tablet_manager.cc:1397] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20260926 03:06:54.544778 8587 raft_consensus.cc:348] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:54.545132 8587 raft_consensus.cc:374] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:54.545279 8587 raft_consensus.cc:729] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8e10deebf684e65a687a362c3ff6f92, State: Initialized, Role: FOLLOWER
I20260926 03:06:54.545630 8587 consensus_queue.cc:260] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:54.548346 8586 tablet_bootstrap.cc:492] T e1b205ad566f4738958ce9ba171a9a1a P 6ce2bd18989b4dfd89abcb754802e625: No bootstrap required, opened a new log
I20260926 03:06:54.548542 8586 ts_tablet_manager.cc:1397] T e1b205ad566f4738958ce9ba171a9a1a P 6ce2bd18989b4dfd89abcb754802e625: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 03:06:54.549690 8586 raft_consensus.cc:348] T e1b205ad566f4738958ce9ba171a9a1a P 6ce2bd18989b4dfd89abcb754802e625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:54.550089 8586 raft_consensus.cc:374] T e1b205ad566f4738958ce9ba171a9a1a P 6ce2bd18989b4dfd89abcb754802e625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:54.550213 8586 raft_consensus.cc:729] T e1b205ad566f4738958ce9ba171a9a1a P 6ce2bd18989b4dfd89abcb754802e625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ce2bd18989b4dfd89abcb754802e625, State: Initialized, Role: FOLLOWER
I20260926 03:06:54.550449 8587 ts_tablet_manager.cc:1428] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260926 03:06:54.550524 8586 consensus_queue.cc:260] T e1b205ad566f4738958ce9ba171a9a1a P 6ce2bd18989b4dfd89abcb754802e625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:54.552065 8588 tablet_bootstrap.cc:492] T e1b205ad566f4738958ce9ba171a9a1a P f7a8e25854b0414e8fd86eb9fc11b327: No bootstrap required, opened a new log
I20260926 03:06:54.552307 8588 ts_tablet_manager.cc:1397] T e1b205ad566f4738958ce9ba171a9a1a P f7a8e25854b0414e8fd86eb9fc11b327: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.004s
I20260926 03:06:54.553187 8588 raft_consensus.cc:348] T e1b205ad566f4738958ce9ba171a9a1a P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:54.553541 8588 raft_consensus.cc:374] T e1b205ad566f4738958ce9ba171a9a1a P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:54.553673 8586 ts_tablet_manager.cc:1428] T e1b205ad566f4738958ce9ba171a9a1a P 6ce2bd18989b4dfd89abcb754802e625: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260926 03:06:54.553681 8588 raft_consensus.cc:729] T e1b205ad566f4738958ce9ba171a9a1a P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7a8e25854b0414e8fd86eb9fc11b327, State: Initialized, Role: FOLLOWER
I20260926 03:06:54.554093 8588 consensus_queue.cc:260] T e1b205ad566f4738958ce9ba171a9a1a P f7a8e25854b0414e8fd86eb9fc11b327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:54.557137 8588 ts_tablet_manager.cc:1428] T e1b205ad566f4738958ce9ba171a9a1a P f7a8e25854b0414e8fd86eb9fc11b327: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260926 03:06:54.563962 8571 ts_tablet_manager.cc:1905] T c52b5de6329a446eaab7dc4bfe4e8cfe P f7a8e25854b0414e8fd86eb9fc11b327: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:54.573096 8571 ts_tablet_manager.cc:1918] T c52b5de6329a446eaab7dc4bfe4e8cfe P f7a8e25854b0414e8fd86eb9fc11b327: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:06:54.574417 8571 log.cc:1196] T c52b5de6329a446eaab7dc4bfe4e8cfe P f7a8e25854b0414e8fd86eb9fc11b327: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-2/wal/wals/c52b5de6329a446eaab7dc4bfe4e8cfe
I20260926 03:06:54.575291 8571 ts_tablet_manager.cc:1939] T c52b5de6329a446eaab7dc4bfe4e8cfe P f7a8e25854b0414e8fd86eb9fc11b327: Deleting consensus metadata
I20260926 03:06:54.577999 7983 catalog_manager.cc:4841] TS f7a8e25854b0414e8fd86eb9fc11b327 (127.0.97.3:34895): tablet c52b5de6329a446eaab7dc4bfe4e8cfe (table test-table [id=243b1d05d7b84862afc95707b3cc3ee0]) successfully deleted
I20260926 03:06:54.662077 8562 raft_consensus.cc:482] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:06:54.662307 8562 raft_consensus.cc:504] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:54.664669 8431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b205ad566f4738958ce9ba171a9a1a" candidate_uuid: "d8e10deebf684e65a687a362c3ff6f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" is_pre_election: true
I20260926 03:06:54.664736 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b205ad566f4738958ce9ba171a9a1a" candidate_uuid: "d8e10deebf684e65a687a362c3ff6f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ce2bd18989b4dfd89abcb754802e625" is_pre_election: true
I20260926 03:06:54.664965 8431 raft_consensus.cc:2454] T e1b205ad566f4738958ce9ba171a9a1a P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8e10deebf684e65a687a362c3ff6f92 in term 0.
I20260926 03:06:54.665017 8156 raft_consensus.cc:2454] T e1b205ad566f4738958ce9ba171a9a1a P 6ce2bd18989b4dfd89abcb754802e625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8e10deebf684e65a687a362c3ff6f92 in term 0.
I20260926 03:06:54.665479 8226 leader_election.cc:302] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8e10deebf684e65a687a362c3ff6f92, f7a8e25854b0414e8fd86eb9fc11b327; no voters:
I20260926 03:06:54.665871 8562 leader_election.cc:288] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ce2bd18989b4dfd89abcb754802e625 (127.0.97.1:32987), f7a8e25854b0414e8fd86eb9fc11b327 (127.0.97.3:34895)
I20260926 03:06:54.666057 8562 raft_consensus.cc:2781] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:06:54.666157 8562 raft_consensus.cc:482] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:06:54.666239 8562 raft_consensus.cc:3037] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:54.669384 8562 raft_consensus.cc:504] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:54.670080 8562 leader_election.cc:288] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92 [CANDIDATE]: Term 1 election: Requested vote from peers 6ce2bd18989b4dfd89abcb754802e625 (127.0.97.1:32987), f7a8e25854b0414e8fd86eb9fc11b327 (127.0.97.3:34895)
I20260926 03:06:54.671125 8156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b205ad566f4738958ce9ba171a9a1a" candidate_uuid: "d8e10deebf684e65a687a362c3ff6f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ce2bd18989b4dfd89abcb754802e625"
I20260926 03:06:54.671363 8156 raft_consensus.cc:3037] T e1b205ad566f4738958ce9ba171a9a1a P 6ce2bd18989b4dfd89abcb754802e625 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:54.674731 8156 raft_consensus.cc:2454] T e1b205ad566f4738958ce9ba171a9a1a P 6ce2bd18989b4dfd89abcb754802e625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8e10deebf684e65a687a362c3ff6f92 in term 1.
I20260926 03:06:54.675397 8224 leader_election.cc:302] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ce2bd18989b4dfd89abcb754802e625, d8e10deebf684e65a687a362c3ff6f92; no voters:
I20260926 03:06:54.675922 8562 raft_consensus.cc:2781] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:54.676102 8562 raft_consensus.cc:686] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92 [term 1 LEADER]: Becoming Leader. State: Replica: d8e10deebf684e65a687a362c3ff6f92, State: Running, Role: LEADER
I20260926 03:06:54.676551 8562 consensus_queue.cc:237] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } }
I20260926 03:06:54.683610 8431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b205ad566f4738958ce9ba171a9a1a" candidate_uuid: "d8e10deebf684e65a687a362c3ff6f92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7a8e25854b0414e8fd86eb9fc11b327"
I20260926 03:06:54.683900 8431 raft_consensus.cc:3037] T e1b205ad566f4738958ce9ba171a9a1a P f7a8e25854b0414e8fd86eb9fc11b327 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:54.686149 7995 catalog_manager.cc:5495] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8e10deebf684e65a687a362c3ff6f92 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "d8e10deebf684e65a687a362c3ff6f92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8e10deebf684e65a687a362c3ff6f92" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 } health_report { overall_health: UNKNOWN } } }
I20260926 03:06:54.687211 8431 raft_consensus.cc:2454] T e1b205ad566f4738958ce9ba171a9a1a P f7a8e25854b0414e8fd86eb9fc11b327 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8e10deebf684e65a687a362c3ff6f92 in term 1.
I20260926 03:06:54.721341 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:54.714476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44082 (local address 127.0.97.2:44011)
0926 03:06:54.714707 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:54.714714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:54.714932 (+ 218us) server_negotiation.cc:407] Connection header received
0926 03:06:54.715115 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:54.715121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:54.715213 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:54.715328 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:06:54.715732 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.716851 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.718904 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.719594 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.719747 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:54.720417 (+ 670us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:06:54.720558 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:06:54.720654 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:54.720796 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 03:06:54.720881 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:06:54.740155 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:54.732752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40453 (local address 127.0.97.3:34895)
0926 03:06:54.733089 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:54.733096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:54.733512 (+ 416us) server_negotiation.cc:407] Connection header received
0926 03:06:54.733759 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:54.733766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:54.733872 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:54.733959 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:06:54.734776 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.735934 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.738051 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.738777 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.738897 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:54.739420 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:06:54.739552 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:06:54.739632 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:54.739776 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 03:06:54.739864 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
I20260926 03:06:54.740541 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:54.732726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34895 (local address 127.0.97.2:40453)
0926 03:06:54.733402 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:54.733422 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:54.733589 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:54.734205 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:54.734210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:54.734224 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:06:54.734572 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:54.734584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.736141 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.736147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:54.737882 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:54.737899 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.738943 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.738949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:54.739128 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:54.739208 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:06:54.739655 (+ 447us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:06:54.739667 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 03:06:54.739750 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 03:06:54.739911 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":394,"threads_started":1}
I20260926 03:06:54.751694 8431 raft_consensus.cc:1261] T e1b205ad566f4738958ce9ba171a9a1a P f7a8e25854b0414e8fd86eb9fc11b327 [term 1 FOLLOWER]: Refusing update from remote peer d8e10deebf684e65a687a362c3ff6f92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:54.752874 8156 raft_consensus.cc:1261] T e1b205ad566f4738958ce9ba171a9a1a P 6ce2bd18989b4dfd89abcb754802e625 [term 1 FOLLOWER]: Refusing update from remote peer d8e10deebf684e65a687a362c3ff6f92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:06:54.752709 8605 consensus_queue.cc:1035] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7a8e25854b0414e8fd86eb9fc11b327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:54.753656 8605 consensus_queue.cc:1035] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ce2bd18989b4dfd89abcb754802e625" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:06:54.811235 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:54.803624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49383 (local address 127.0.97.2:44011)
0926 03:06:54.803806 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:54.803813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:54.804388 (+ 575us) server_negotiation.cc:407] Connection header received
0926 03:06:54.804688 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:54.804693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:54.804776 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:54.804837 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:06:54.805605 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.806741 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.809043 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.809728 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.809853 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:54.810372 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:06:54.810544 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:06:54.810628 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:54.810822 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 03:06:54.810916 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 03:06:54.811367 8613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:54.803513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44011 (local address 127.0.97.3:49383)
0926 03:06:54.804274 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:54.804294 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:54.804520 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:54.805036 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:54.805041 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:54.805059 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:06:54.805399 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:54.805412 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.806889 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.806895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:54.808895 (+ 2000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:54.808913 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.809907 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.809913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:54.810112 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:54.810165 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:06:54.810669 (+ 504us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:06:54.810682 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 03:06:54.810782 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 03:06:54.810942 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":490,"threads_started":1}
I20260926 03:06:54.870364 7994 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:06:54.870684 7994 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:06:54.874756 7994 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 665ebc9724d34582a9f1ba3490f657ab: Sending DeleteTablet for 3 replicas of tablet e1b205ad566f4738958ce9ba171a9a1a
I20260926 03:06:54.875737 8133 tablet_service.cc:1514] Processing DeleteTablet for tablet e1b205ad566f4738958ce9ba171a9a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:54 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49838
I20260926 03:06:54.876063 8570 tablet_replica.cc:330] T e1b205ad566f4738958ce9ba171a9a1a P 6ce2bd18989b4dfd89abcb754802e625: stopping tablet replica
I20260926 03:06:54.876173 8272 tablet_service.cc:1514] Processing DeleteTablet for tablet e1b205ad566f4738958ce9ba171a9a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:54 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51228
I20260926 03:06:54.876266 8570 raft_consensus.cc:2229] T e1b205ad566f4738958ce9ba171a9a1a P 6ce2bd18989b4dfd89abcb754802e625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:06:54.876451 8572 tablet_replica.cc:330] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92: stopping tablet replica
I20260926 03:06:54.876487 8570 raft_consensus.cc:2258] T e1b205ad566f4738958ce9ba171a9a1a P 6ce2bd18989b4dfd89abcb754802e625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:54.876621 8572 raft_consensus.cc:2229] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:06:54.876979 8572 raft_consensus.cc:2258] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:54.877142 8411 tablet_service.cc:1514] Processing DeleteTablet for tablet e1b205ad566f4738958ce9ba171a9a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:06:54 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57562
I20260926 03:06:54.877458 8571 tablet_replica.cc:330] T e1b205ad566f4738958ce9ba171a9a1a P f7a8e25854b0414e8fd86eb9fc11b327: stopping tablet replica
I20260926 03:06:54.877655 8571 raft_consensus.cc:2229] T e1b205ad566f4738958ce9ba171a9a1a P f7a8e25854b0414e8fd86eb9fc11b327 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:06:54.877879 8571 raft_consensus.cc:2258] T e1b205ad566f4738958ce9ba171a9a1a P f7a8e25854b0414e8fd86eb9fc11b327 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:06:54.879554 8571 ts_tablet_manager.cc:1905] T e1b205ad566f4738958ce9ba171a9a1a P f7a8e25854b0414e8fd86eb9fc11b327: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:54.883272 8572 ts_tablet_manager.cc:1905] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:54.887514 8571 ts_tablet_manager.cc:1918] T e1b205ad566f4738958ce9ba171a9a1a P f7a8e25854b0414e8fd86eb9fc11b327: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:06:54.887636 8570 ts_tablet_manager.cc:1905] T e1b205ad566f4738958ce9ba171a9a1a P 6ce2bd18989b4dfd89abcb754802e625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:06:54.887701 8571 log.cc:1196] T e1b205ad566f4738958ce9ba171a9a1a P f7a8e25854b0414e8fd86eb9fc11b327: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-2/wal/wals/e1b205ad566f4738958ce9ba171a9a1a
I20260926 03:06:54.888151 8571 ts_tablet_manager.cc:1939] T e1b205ad566f4738958ce9ba171a9a1a P f7a8e25854b0414e8fd86eb9fc11b327: Deleting consensus metadata
I20260926 03:06:54.889369 7983 catalog_manager.cc:4841] TS f7a8e25854b0414e8fd86eb9fc11b327 (127.0.97.3:34895): tablet e1b205ad566f4738958ce9ba171a9a1a (table test-table [id=c9061620f39f45299652fc8a76946f49]) successfully deleted
I20260926 03:06:54.892589 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 8067
I20260926 03:06:54.897750 8572 ts_tablet_manager.cc:1918] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:06:54.897917 8572 log.cc:1196] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-1/wal/wals/e1b205ad566f4738958ce9ba171a9a1a
I20260926 03:06:54.898343 8572 ts_tablet_manager.cc:1939] T e1b205ad566f4738958ce9ba171a9a1a P d8e10deebf684e65a687a362c3ff6f92: Deleting consensus metadata
I20260926 03:06:54.899675 7983 catalog_manager.cc:4841] TS d8e10deebf684e65a687a362c3ff6f92 (127.0.97.2:44011): tablet e1b205ad566f4738958ce9ba171a9a1a (table test-table [id=c9061620f39f45299652fc8a76946f49]) successfully deleted
I20260926 03:06:54.902066 8570 ts_tablet_manager.cc:1918] T e1b205ad566f4738958ce9ba171a9a1a P 6ce2bd18989b4dfd89abcb754802e625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:06:54.902379 8570 log.cc:1196] T e1b205ad566f4738958ce9ba171a9a1a P 6ce2bd18989b4dfd89abcb754802e625: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391950058021-388-0/minicluster-data/ts-0/wal/wals/e1b205ad566f4738958ce9ba171a9a1a
I20260926 03:06:54.903218 8570 ts_tablet_manager.cc:1939] T e1b205ad566f4738958ce9ba171a9a1a P 6ce2bd18989b4dfd89abcb754802e625: Deleting consensus metadata
I20260926 03:06:54.904688 7981 catalog_manager.cc:4841] TS 6ce2bd18989b4dfd89abcb754802e625 (127.0.97.1:32987): tablet e1b205ad566f4738958ce9ba171a9a1a (table test-table [id=c9061620f39f45299652fc8a76946f49]) successfully deleted
Sep 26 03:06:54 dist-test-slave-hmbw krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260926 03:06:54.926472 8598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:54.893844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45966 (local address 127.0.97.1:32987)
0926 03:06:54.894057 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:54.894064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:54.899915 (+ 5851us) server_negotiation.cc:407] Connection header received
0926 03:06:54.900216 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:54.900222 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:54.900327 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:54.900402 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:54.902266 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.903544 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.905488 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:54.906211 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:54.906372 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:54.918534 (+ 12162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:54.918557 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:54.918585 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:54.918645 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:54.922849 (+ 4204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:54.923876 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:54.923888 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:54.923892 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:54.924084 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:54.924594 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:54.924600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:54.924603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:54.925103 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:54.925457 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:54.925784 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 03:06:54.926031 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:06:54.927374 8198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:55.058665 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 8067
I20260926 03:06:55.085628 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 8207
Sep 26 03:06:55 dist-test-slave-hmbw krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260926 03:06:55.101642 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:55.086379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44098 (local address 127.0.97.2:44011)
0926 03:06:55.086589 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:55.086594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:55.086612 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:06:55.086789 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:55.086793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:55.086855 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:55.086939 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:55.088411 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:55.089101 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:55.090482 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:55.090913 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:55.091015 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:55.094979 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:55.094994 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:55.095010 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:55.095053 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:55.099114 (+ 4061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:55.099885 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:55.099893 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:55.099895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:55.100035 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:55.100350 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:55.100355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:55.100357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:55.100718 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:55.100993 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:55.101235 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 03:06:55.101402 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:06:55.102377 8337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:55.223280 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 8207
I20260926 03:06:55.241748 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 8346
Sep 26 03:06:55 dist-test-slave-hmbw krb5kdc[7951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260926 03:06:55.256582 8619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:55.242638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45920 (local address 127.0.97.3:34895)
0926 03:06:55.243022 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:55.243026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:55.243042 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:06:55.243093 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:55.243097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:55.243154 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:55.243238 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:55.244473 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:55.245217 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:55.246425 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:55.247016 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:55.247112 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:55.251155 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:55.251171 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:55.251189 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:55.251240 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:55.253990 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:55.254780 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:55.254789 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:55.254791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:55.254948 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:55.255280 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:55.255284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:55.255286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:55.255647 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:55.255870 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:55.256093 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 03:06:55.256287 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":193,"threads_started":1}
I20260926 03:06:55.257270 8476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:55.398521 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 8346
I20260926 03:06:55.419373 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 7964
I20260926 03:06:55.420608 8036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:06:55.532130 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 7964
2026-09-26T03:06:55Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5801 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:06:55 dist-test-slave-hmbw krb5kdc[8625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:06:55 dist-test-slave-hmbw krb5kdc[8625](info): set up 2 sockets
Sep 26 03:06:55 dist-test-slave-hmbw krb5kdc[8625](info): commencing operation
krb5kdc: starting...
W20260926 03:06:56.555369 388 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.008s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:06:56 dist-test-slave-hmbw krb5kdc[8625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:06:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:06:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:06:56.777964 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40557
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:57.101452 8638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:57.101763 8638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:57.101851 8638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:57.110750 8638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:06:57.110872 8638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:57.110950 8638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:57.111027 8638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:06:57.111089 8638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:06:57.121392 8638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:57.122776 8638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:57.124132 8638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:57.137691 8646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:57.137929 8643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:57.137912 8644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:57.139459 8638 server_base.cc:1034] running on GCE node
I20260926 03:06:57.140516 8638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:57.142031 8638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:57.143257 8638 hybrid_clock.cc:648] HybridClock initialized: now 1790392017143202 us; error 51 us; skew 500 ppm
I20260926 03:06:57.143617 8638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T03:06:57Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1676 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:06:57 dist-test-slave-hmbw krb5kdc[8652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:06:57 dist-test-slave-hmbw krb5kdc[8652](info): set up 2 sockets
Sep 26 03:06:57 dist-test-slave-hmbw krb5kdc[8652](info): commencing operation
krb5kdc: starting...
W20260926 03:06:58.215325 388 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:06:58 dist-test-slave-hmbw krb5kdc[8652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:06:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:06:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:06:58.435415 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42423
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42423
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:58.760474 8665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:58.760778 8665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:58.760898 8665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:58.768442 8665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:06:58.768563 8665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:58.768618 8665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:58.768697 8665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:06:58.768751 8665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:06:58.778791 8665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:58.780139 8665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:58.781491 8665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:58.793437 8673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:58.793618 8671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:58.793624 8670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:58.794988 8665 server_base.cc:1034] running on GCE node
I20260926 03:06:58.795873 8665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:58.797529 8665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:58.798779 8665 hybrid_clock.cc:648] HybridClock initialized: now 1790392018798727 us; error 44 us; skew 500 ppm
I20260926 03:06:58.799266 8665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:58 dist-test-slave-hmbw krb5kdc[8652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:58.802608 8665 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:06:58.804317 8665 webserver.cc:460] Webserver started at http://127.0.97.62:36863/ using document root <none> and password file <none>
I20260926 03:06:58.805047 8665 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:58.805195 8665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:58.805579 8665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:58.808831 8665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "b66e293775084aada1d1b32ec78ca9fa"
format_stamp: "Formatted at 2026-09-26 03:06:58 on dist-test-slave-hmbw"
server_key: "979c684dfd4c541049ba0ef2f0da2577"
server_key_iv: "49f2d131a414696052353a9f26ae0927"
server_key_version: "encryptionkey@0"
I20260926 03:06:58.809612 8665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "b66e293775084aada1d1b32ec78ca9fa"
format_stamp: "Formatted at 2026-09-26 03:06:58 on dist-test-slave-hmbw"
server_key: "979c684dfd4c541049ba0ef2f0da2577"
server_key_iv: "49f2d131a414696052353a9f26ae0927"
server_key_version: "encryptionkey@0"
I20260926 03:06:58.815219 8665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:06:58.819276 8680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:58.820971 8665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:06:58.821185 8665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "b66e293775084aada1d1b32ec78ca9fa"
format_stamp: "Formatted at 2026-09-26 03:06:58 on dist-test-slave-hmbw"
server_key: "979c684dfd4c541049ba0ef2f0da2577"
server_key_iv: "49f2d131a414696052353a9f26ae0927"
server_key_version: "encryptionkey@0"
I20260926 03:06:58.821403 8665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:58.848789 8665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:58.853295 8665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:58.853600 8665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:58.871392 8665 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42423
I20260926 03:06:58.871429 8731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42423 every 8 connection(s)
I20260926 03:06:58.872800 8665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:06:58.876971 8732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:06:58.876951 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 8665
I20260926 03:06:58.877166 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:06:58.877976 388 external_mini_cluster.cc:1428] Setting key bdb64267d7667e3a639024d8daf00f5d
I20260926 03:06:58.891611 8732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa: Bootstrap starting.
Sep 26 03:06:58 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:06:58.895525 8732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa: Neither blocks nor log segments found. Creating new log.
I20260926 03:06:58.896767 8732 log.cc:824] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa: Log is configured to *not* fsync() on all Append() calls
I20260926 03:06:58.900492 8732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa: No bootstrap required, opened a new log
I20260926 03:06:58.904736 8735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:58.880180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33058 (local address 127.0.97.62:42423)
0926 03:06:58.881194 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:58.881211 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:58.881257 (+ 46us) server_negotiation.cc:407] Connection header received
0926 03:06:58.883414 (+ 2157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:58.883449 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:58.883964 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:58.884344 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:58.886187 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:58.887575 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:58.889511 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:58.890291 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:58.890459 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:58.895259 (+ 4800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:58.895269 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:58.895285 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:58.895328 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:58.899027 (+ 3699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:58.900036 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:58.900044 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:58.900051 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:58.900238 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:58.900900 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:58.900906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:58.900909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:58.901729 (+ 820us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:58.902016 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:58.902722 (+ 706us) server_negotiation.cc:299] Negotiation successful
0926 03:06:58.903221 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":352,"threads_started":1}
I20260926 03:06:58.909835 8732 raft_consensus.cc:348] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66e293775084aada1d1b32ec78ca9fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42423 } }
I20260926 03:06:58.910234 8732 raft_consensus.cc:374] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:06:58.910324 8732 raft_consensus.cc:729] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b66e293775084aada1d1b32ec78ca9fa, State: Initialized, Role: FOLLOWER
I20260926 03:06:58.911020 8732 consensus_queue.cc:260] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66e293775084aada1d1b32ec78ca9fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42423 } }
I20260926 03:06:58.911283 8732 raft_consensus.cc:388] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:06:58.911444 8732 raft_consensus.cc:482] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:06:58.911614 8732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:06:58.913780 8732 raft_consensus.cc:504] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66e293775084aada1d1b32ec78ca9fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42423 } }
I20260926 03:06:58.914328 8732 leader_election.cc:302] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b66e293775084aada1d1b32ec78ca9fa; no voters:
I20260926 03:06:58.915007 8732 leader_election.cc:288] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:06:58.915129 8737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:06:58.915773 8737 raft_consensus.cc:686] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [term 1 LEADER]: Becoming Leader. State: Replica: b66e293775084aada1d1b32ec78ca9fa, State: Running, Role: LEADER
I20260926 03:06:58.916425 8737 consensus_queue.cc:237] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66e293775084aada1d1b32ec78ca9fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42423 } }
I20260926 03:06:58.917061 8732 sys_catalog.cc:564] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:06:58.919371 8739 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader b66e293775084aada1d1b32ec78ca9fa. Latest consensus state: current_term: 1 leader_uuid: "b66e293775084aada1d1b32ec78ca9fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66e293775084aada1d1b32ec78ca9fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42423 } } }
I20260926 03:06:58.919828 8739 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:58.920370 8738 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b66e293775084aada1d1b32ec78ca9fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66e293775084aada1d1b32ec78ca9fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42423 } } }
I20260926 03:06:58.920674 8738 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [sys.catalog]: This master's current role is: LEADER
I20260926 03:06:58.921033 8744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:06:58.927627 8744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:06:58.937575 8744 catalog_manager.cc:1347] Generated new cluster ID: bd18aa6295d941a5966590561c8e5f34
I20260926 03:06:58.937702 8744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:06:58.958297 8744 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:06:58.960188 8744 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:06:58.966990 8744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa: Generated new TSK 0
I20260926 03:06:58.967908 8744 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:06:59.039930 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42423
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:59.402570 8758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:59.402882 8758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:59.402989 8758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:59.410454 8758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:59.410706 8758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:59.410884 8758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:06:59.421288 8758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:59.422706 8758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:59.424345 8758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:59.438102 8764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:59.438124 8766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:59.438174 8763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:59.439004 8758 server_base.cc:1034] running on GCE node
I20260926 03:06:59.439670 8758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:59.440687 8758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:59.441915 8758 hybrid_clock.cc:648] HybridClock initialized: now 1790392019441848 us; error 60 us; skew 500 ppm
I20260926 03:06:59.442262 8758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:59 dist-test-slave-hmbw krb5kdc[8652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:59.445672 8758 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:06:59.447431 8758 webserver.cc:460] Webserver started at http://127.0.97.1:38335/ using document root <none> and password file <none>
I20260926 03:06:59.448165 8758 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:59.448295 8758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:59.448623 8758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:59.451735 8758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/data/instance:
uuid: "d321135811ab4631b44a47f2df504e0e"
format_stamp: "Formatted at 2026-09-26 03:06:59 on dist-test-slave-hmbw"
server_key: "e743a3bdf8e68a12f349f2cf5fb763ea"
server_key_iv: "a2d997f6905e6a5833c8496f7beb1dfc"
server_key_version: "encryptionkey@0"
I20260926 03:06:59.452533 8758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d321135811ab4631b44a47f2df504e0e"
format_stamp: "Formatted at 2026-09-26 03:06:59 on dist-test-slave-hmbw"
server_key: "e743a3bdf8e68a12f349f2cf5fb763ea"
server_key_iv: "a2d997f6905e6a5833c8496f7beb1dfc"
server_key_version: "encryptionkey@0"
I20260926 03:06:59.458201 8758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:06:59.462118 8773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:59.463760 8758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:06:59.463980 8758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "d321135811ab4631b44a47f2df504e0e"
format_stamp: "Formatted at 2026-09-26 03:06:59 on dist-test-slave-hmbw"
server_key: "e743a3bdf8e68a12f349f2cf5fb763ea"
server_key_iv: "a2d997f6905e6a5833c8496f7beb1dfc"
server_key_version: "encryptionkey@0"
I20260926 03:06:59.464267 8758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:06:59.494887 8758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:06:59.499351 8758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:06:59.499660 8758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:06:59.500646 8758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:06:59.502071 8758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:06:59.502152 8758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:59.502269 8758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:06:59.502331 8758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:59.534641 8758 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44599
I20260926 03:06:59.534709 8885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44599 every 8 connection(s)
I20260926 03:06:59.535980 8758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:06:59.536490 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 8758
I20260926 03:06:59.536697 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:06:59.537546 388 external_mini_cluster.cc:1428] Setting key cd698997d2cca038d963d8e5759d49c0
Sep 26 03:06:59 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:06:59.558694 8888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:59.539331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53869 (local address 127.0.97.62:42423)
0926 03:06:59.539778 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0926 03:06:59.539784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:06:59.541067 (+ 1283us) server_negotiation.cc:407] Connection header received
0926 03:06:59.543500 (+ 2433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:06:59.543504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:06:59.543578 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:06:59.543672 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:06:59.545719 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:59.546527 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:59.548710 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:06:59.549114 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:06:59.549233 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:59.553025 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:06:59.553031 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:06:59.553034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:06:59.553071 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:06:59.555611 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:59.556389 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:59.556393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:59.556396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:59.556515 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:06:59.556954 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:06:59.556958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:06:59.556960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:06:59.557226 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:06:59.557352 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 03:06:59.558079 (+ 727us) server_negotiation.cc:299] Negotiation successful
0926 03:06:59.558307 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":221,"threads_started":1}
I20260926 03:06:59.559820 8889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:06:59.539992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.1:53869)
0926 03:06:59.540833 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0926 03:06:59.540890 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 03:06:59.543159 (+ 2269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:06:59.543989 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:06:59.544001 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:06:59.544694 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:06:59.545511 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:59.545536 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:59.546717 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:59.546721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:59.548543 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:06:59.548561 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:06:59.549361 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:06:59.549368 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:06:59.549649 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:06:59.550341 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:06:59.550363 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:06:59.552753 (+ 2390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:06:59.555791 (+ 3038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:59.555801 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:59.555820 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:59.556263 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:59.556621 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:06:59.556626 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:06:59.556630 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:06:59.556847 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:06:59.557348 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:06:59.557357 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:06:59.557716 (+ 359us) client_negotiation.cc:769] Sending connection context
0926 03:06:59.557958 (+ 242us) client_negotiation.cc:240] Negotiation successful
0926 03:06:59.558551 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":315,"threads_started":1}
I20260926 03:06:59.561527 8886 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 03:06:59.561909 8886 heartbeater.cc:464] Registering TS with master...
I20260926 03:06:59.562808 8886 heartbeater.cc:511] Master 127.0.97.62:42423 requested a full tablet report, sending...
I20260926 03:06:59.565078 8697 ts_manager.cc:194] Registered new tserver with Master: d321135811ab4631b44a47f2df504e0e (127.0.97.1:44599)
I20260926 03:06:59.566583 8697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:06:59.590126 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42423
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:06:59.915876 8892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:06:59.916183 8892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:06:59.916306 8892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:06:59.923820 8892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:06:59.923959 8892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:06:59.924103 8892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:06:59.933961 8892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:06:59.935359 8892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:06:59.936650 8892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:06:59.950666 8898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:59.950707 8897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:06:59.950736 8900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:06:59.951390 8892 server_base.cc:1034] running on GCE node
I20260926 03:06:59.952039 8892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:06:59.952993 8892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:06:59.954198 8892 hybrid_clock.cc:648] HybridClock initialized: now 1790392019954150 us; error 39 us; skew 500 ppm
I20260926 03:06:59.954542 8892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:06:59 dist-test-slave-hmbw krb5kdc[8652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:06:59.968499 8892 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:06:59.970181 8892 webserver.cc:460] Webserver started at http://127.0.97.2:38029/ using document root <none> and password file <none>
I20260926 03:06:59.970947 8892 fs_manager.cc:359] Metadata directory not provided
I20260926 03:06:59.971071 8892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:06:59.971366 8892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:06:59.974284 8892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/data/instance:
uuid: "65ad33ce1ce94b50a878adc4d27a1cd8"
format_stamp: "Formatted at 2026-09-26 03:06:59 on dist-test-slave-hmbw"
server_key: "e2e58f0dad214ad82e32ce5d1dcc2f25"
server_key_iv: "9f36c4962b61b994d0da7501673d3ff8"
server_key_version: "encryptionkey@0"
I20260926 03:06:59.975078 8892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/wal/instance:
uuid: "65ad33ce1ce94b50a878adc4d27a1cd8"
format_stamp: "Formatted at 2026-09-26 03:06:59 on dist-test-slave-hmbw"
server_key: "e2e58f0dad214ad82e32ce5d1dcc2f25"
server_key_iv: "9f36c4962b61b994d0da7501673d3ff8"
server_key_version: "encryptionkey@0"
I20260926 03:06:59.980372 8892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:06:59.983923 8907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:06:59.985381 8892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:06:59.985553 8892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "65ad33ce1ce94b50a878adc4d27a1cd8"
format_stamp: "Formatted at 2026-09-26 03:06:59 on dist-test-slave-hmbw"
server_key: "e2e58f0dad214ad82e32ce5d1dcc2f25"
server_key_iv: "9f36c4962b61b994d0da7501673d3ff8"
server_key_version: "encryptionkey@0"
I20260926 03:06:59.985738 8892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:07:00.012447 8892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:07:00.018117 8892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:07:00.018684 8892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:07:00.020478 8892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:07:00.022603 8892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:07:00.022706 8892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:00.023130 8892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:07:00.023468 8892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:00.058224 8892 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41937
I20260926 03:07:00.058265 9019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41937 every 8 connection(s)
I20260926 03:07:00.059644 8892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:07:00.061396 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 8892
I20260926 03:07:00.061554 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:07:00.062083 388 external_mini_cluster.cc:1428] Setting key c8cfa527870b60f20418e47737e6050f
Sep 26 03:07:00 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:07:00.084479 9022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:00.062617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44429 (local address 127.0.97.62:42423)
0926 03:07:00.063076 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:00.063083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:00.064309 (+ 1226us) server_negotiation.cc:407] Connection header received
0926 03:07:00.066951 (+ 2642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:00.066956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:00.067031 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:00.067128 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:00.069259 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:00.070105 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:00.072382 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:00.073104 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:00.073250 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:00.077178 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:00.077185 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:00.077190 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:07:00.077236 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:00.080447 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:07:00.081374 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:07:00.081380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:07:00.081384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:07:00.081547 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:07:00.082075 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:07:00.082080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:07:00.082084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:07:00.082489 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:07:00.082698 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:00.083801 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0926 03:07:00.084071 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":216,"threads_started":1}
I20260926 03:07:00.085498 9023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:00.063275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.2:44429)
0926 03:07:00.064130 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:00.064187 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:00.066602 (+ 2415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:00.067479 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:00.067491 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:00.068195 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:00.069036 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:00.069065 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:00.070276 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:00.070281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:00.072168 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:00.072184 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:00.073374 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:00.073380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:00.073593 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:00.074308 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:00.074334 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:00.076898 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:07:00.080645 (+ 3747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:07:00.080654 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:07:00.080669 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:07:00.081222 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:07:00.081697 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:07:00.081703 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:07:00.081707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:07:00.081940 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:07:00.082689 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:07:00.082696 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:07:00.083109 (+ 413us) client_negotiation.cc:769] Sending connection context
0926 03:07:00.083663 (+ 554us) client_negotiation.cc:240] Negotiation successful
0926 03:07:00.084219 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":320,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 03:07:00.087280 9020 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42423
I20260926 03:07:00.087611 9020 heartbeater.cc:464] Registering TS with master...
I20260926 03:07:00.088415 9020 heartbeater.cc:511] Master 127.0.97.62:42423 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 03:07:00.090274 8697 ts_manager.cc:194] Registered new tserver with Master: 65ad33ce1ce94b50a878adc4d27a1cd8 (127.0.97.2:41937)
I20260926 03:07:00.091419 8697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44429
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:07:00.118417 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42423
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:07:00.453389 9026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:07:00.453706 9026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:07:00.453840 9026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:07:00.461220 9026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:07:00.461362 9026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:07:00.461503 9026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:07:00.472029 9026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:07:00.473693 9026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:07:00.475144 9026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:07:00.489504 9031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:00.489411 9032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:00.489444 9034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:07:00.492409 9026 server_base.cc:1034] running on GCE node
I20260926 03:07:00.493103 9026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:07:00.494189 9026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:07:00.495414 9026 hybrid_clock.cc:648] HybridClock initialized: now 1790392020495349 us; error 63 us; skew 500 ppm
I20260926 03:07:00.495741 9026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:07:00 dist-test-slave-hmbw krb5kdc[8652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:07:00.499336 9026 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:07:00.501080 9026 webserver.cc:460] Webserver started at http://127.0.97.3:33577/ using document root <none> and password file <none>
I20260926 03:07:00.501843 9026 fs_manager.cc:359] Metadata directory not provided
I20260926 03:07:00.501967 9026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:07:00.502302 9026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:07:00.505749 9026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/data/instance:
uuid: "9ea4e0742ee84d818ef25c759ae54990"
format_stamp: "Formatted at 2026-09-26 03:07:00 on dist-test-slave-hmbw"
server_key: "287e089aab5309877883063f215345b1"
server_key_iv: "503a2083fe149441191255a58435be1f"
server_key_version: "encryptionkey@0"
I20260926 03:07:00.506762 9026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9ea4e0742ee84d818ef25c759ae54990"
format_stamp: "Formatted at 2026-09-26 03:07:00 on dist-test-slave-hmbw"
server_key: "287e089aab5309877883063f215345b1"
server_key_iv: "503a2083fe149441191255a58435be1f"
server_key_version: "encryptionkey@0"
I20260926 03:07:00.512439 9026 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:07:00.516376 9041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:00.518105 9026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:07:00.518296 9026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "9ea4e0742ee84d818ef25c759ae54990"
format_stamp: "Formatted at 2026-09-26 03:07:00 on dist-test-slave-hmbw"
server_key: "287e089aab5309877883063f215345b1"
server_key_iv: "503a2083fe149441191255a58435be1f"
server_key_version: "encryptionkey@0"
I20260926 03:07:00.518548 9026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:07:00.538674 9026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:07:00.543198 9026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:07:00.543473 9026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:07:00.544483 9026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:07:00.546007 9026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:07:00.546082 9026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:00.546188 9026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:07:00.546233 9026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:00.570833 8886 heartbeater.cc:503] Master 127.0.97.62:42423 was elected leader, sending a full tablet report...
I20260926 03:07:00.578429 9026 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41513
I20260926 03:07:00.578473 9153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41513 every 8 connection(s)
I20260926 03:07:00.579702 9026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:07:00.580574 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 9026
I20260926 03:07:00.580725 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:07:00.581216 388 external_mini_cluster.cc:1428] Setting key 025422b0817923ad52a92c150b796f9b
Sep 26 03:07:00 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:07:00.600827 9022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:00.582221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51671 (local address 127.0.97.62:42423)
0926 03:07:00.582372 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:00.582376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:00.583822 (+ 1446us) server_negotiation.cc:407] Connection header received
0926 03:07:00.586006 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:00.586011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:00.586110 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:00.586193 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:00.587809 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:00.588530 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:00.590126 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:00.590593 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:00.590717 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:00.594415 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:00.594422 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:00.594426 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:07:00.594469 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:00.597608 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:07:00.598477 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:07:00.598482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:07:00.598484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:07:00.598648 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:07:00.599130 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:07:00.599134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:07:00.599136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:07:00.599469 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:07:00.599590 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:00.600383 (+ 793us) server_negotiation.cc:299] Negotiation successful
0926 03:07:00.600592 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 03:07:00.601848 9156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:00.582890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.3:51671)
0926 03:07:00.583668 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:00.583705 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:00.585677 (+ 1972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:00.586472 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:00.586480 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:00.587044 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:00.587628 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:00.587648 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:00.588686 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:00.588690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:00.589975 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:00.589989 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:00.590837 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:00.590844 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:00.591071 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:00.591728 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:00.591756 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:00.594128 (+ 2372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:07:00.597773 (+ 3645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:07:00.597781 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:07:00.597797 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:07:00.598336 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:07:00.598756 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:07:00.598759 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:07:00.598762 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:07:00.598963 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:07:00.599618 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:07:00.599624 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:07:00.599989 (+ 365us) client_negotiation.cc:769] Sending connection context
0926 03:07:00.600252 (+ 263us) client_negotiation.cc:240] Negotiation successful
0926 03:07:00.600742 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":289,"threads_started":1}
I20260926 03:07:00.603549 9154 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42423
I20260926 03:07:00.603866 9154 heartbeater.cc:464] Registering TS with master...
I20260926 03:07:00.604647 9154 heartbeater.cc:511] Master 127.0.97.62:42423 requested a full tablet report, sending...
I20260926 03:07:00.606848 8697 ts_manager.cc:194] Registered new tserver with Master: 9ea4e0742ee84d818ef25c759ae54990 (127.0.97.3:41513)
I20260926 03:07:00.607718 8697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51671
I20260926 03:07:00.609984 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260926 03:07:00.755012 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 8758
Sep 26 03:07:00 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:07:00.772756 9164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:00.756180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52388 (local address 127.0.97.1:44599)
0926 03:07:00.756844 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:00.756854 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:00.756886 (+ 32us) server_negotiation.cc:407] Connection header received
0926 03:07:00.756946 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:00.756953 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:00.757161 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:00.757325 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:00.758741 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:00.759933 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:00.761428 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:00.762057 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:00.762175 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:00.766711 (+ 4536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:00.766728 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:00.766743 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:07:00.766793 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:00.769755 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:07:00.770651 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:07:00.770658 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:07:00.770661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:07:00.770791 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:07:00.771183 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:07:00.771186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:07:00.771188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:07:00.771539 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:07:00.771764 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:00.772150 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 03:07:00.772430 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":336,"threads_started":1}
I20260926 03:07:00.774149 8881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:07:00.857723 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 8758
I20260926 03:07:00.871692 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 8892
Sep 26 03:07:00 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:07:00.889329 9166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:00.872592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40398 (local address 127.0.97.2:41937)
0926 03:07:00.873172 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:00.873182 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:00.873215 (+ 33us) server_negotiation.cc:407] Connection header received
0926 03:07:00.873283 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:00.873292 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:00.873580 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:00.873746 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:00.874906 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:00.876570 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:00.877876 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:00.878855 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:00.878995 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:00.883201 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:00.883225 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:00.883245 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:07:00.883320 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:00.886262 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:07:00.887318 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:07:00.887329 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:07:00.887332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:07:00.887482 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:07:00.887839 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:07:00.887843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:07:00.887845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:07:00.888241 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:07:00.888522 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:00.888740 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 03:07:00.888977 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":277,"threads_started":1}
I20260926 03:07:00.890769 9015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:07:00.973097 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 8892
I20260926 03:07:00.986332 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 9026
Sep 26 03:07:00 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:07:01.003935 9168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:00.987608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34188 (local address 127.0.97.3:41513)
0926 03:07:00.988300 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:00.988307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:00.988337 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:07:00.988407 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:00.988414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:00.988616 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:00.988775 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:00.990123 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:00.991248 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:00.992537 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:00.993126 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:00.993248 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:00.997896 (+ 4648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:00.997912 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:00.997929 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:07:00.997977 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:01.000907 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:07:01.001700 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:07:01.001709 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:07:01.001711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:07:01.001852 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:07:01.002243 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:07:01.002247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:07:01.002249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:07:01.002684 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:07:01.003007 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:01.003304 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 03:07:01.003591 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":402,"threads_started":1}
I20260926 03:07:01.005368 9149 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:07:01.092918 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 9026
I20260926 03:07:01.106426 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 8665
I20260926 03:07:01.107815 8727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:07:01.212450 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 8665
I20260926 03:07:01.230345 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42423
--webserver_interface=127.0.97.62
--webserver_port=36863
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42423
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:07:01.578713 9171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:07:01.579025 9171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:07:01.579125 9171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:07:01.586407 9171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:07:01.586607 9171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:07:01.586687 9171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:07:01.586745 9171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:07:01.586789 9171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:07:01.597023 9171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=36863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:07:01.598368 9171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:07:01.599740 9171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:07:01.611341 9177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:01.611393 9179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:01.611804 9176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:07:01.612083 9171 server_base.cc:1034] running on GCE node
I20260926 03:07:01.612859 9171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:07:01.614317 9171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:07:01.615573 9171 hybrid_clock.cc:648] HybridClock initialized: now 1790392021615513 us; error 51 us; skew 500 ppm
I20260926 03:07:01.615969 9171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:07:01 dist-test-slave-hmbw krb5kdc[8652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:07:01.619225 9171 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:07:01.620786 9171 webserver.cc:460] Webserver started at http://127.0.97.62:36863/ using document root <none> and password file <none>
I20260926 03:07:01.621538 9171 fs_manager.cc:359] Metadata directory not provided
I20260926 03:07:01.621683 9171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:07:01.627671 9171 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 03:07:01.630853 9186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:01.632545 9171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:07:01.632725 9171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "b66e293775084aada1d1b32ec78ca9fa"
format_stamp: "Formatted at 2026-09-26 03:06:58 on dist-test-slave-hmbw"
server_key: "979c684dfd4c541049ba0ef2f0da2577"
server_key_iv: "49f2d131a414696052353a9f26ae0927"
server_key_version: "encryptionkey@0"
I20260926 03:07:01.633705 9171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:07:01.666973 9171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:07:01.671514 9171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:07:01.671790 9171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:07:01.690096 9171 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42423
I20260926 03:07:01.690119 9237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42423 every 8 connection(s)
I20260926 03:07:01.691671 9171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:07:01.693190 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 9171
I20260926 03:07:01.693940 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:44599
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:42423
--webserver_port=38335
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:07:01.698941 9238 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:07:01.702379 9238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa: Bootstrap starting.
I20260926 03:07:01.721128 9238 log.cc:824] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa: Log is configured to *not* fsync() on all Append() calls
I20260926 03:07:01.730532 9238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:07:01.731074 9238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa: Bootstrap complete.
I20260926 03:07:01.740764 9238 raft_consensus.cc:348] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66e293775084aada1d1b32ec78ca9fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42423 } }
I20260926 03:07:01.741662 9238 raft_consensus.cc:729] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b66e293775084aada1d1b32ec78ca9fa, State: Initialized, Role: FOLLOWER
I20260926 03:07:01.742365 9238 consensus_queue.cc:260] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66e293775084aada1d1b32ec78ca9fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42423 } }
I20260926 03:07:01.742681 9238 raft_consensus.cc:388] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:07:01.742796 9238 raft_consensus.cc:482] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:07:01.742954 9238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:07:01.746933 9238 raft_consensus.cc:504] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66e293775084aada1d1b32ec78ca9fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42423 } }
I20260926 03:07:01.747473 9238 leader_election.cc:302] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b66e293775084aada1d1b32ec78ca9fa; no voters:
I20260926 03:07:01.748039 9238 leader_election.cc:288] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 03:07:01.748229 9242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [term 2 FOLLOWER]: Leader election won for term 2
I20260926 03:07:01.749161 9242 raft_consensus.cc:686] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [term 2 LEADER]: Becoming Leader. State: Replica: b66e293775084aada1d1b32ec78ca9fa, State: Running, Role: LEADER
I20260926 03:07:01.749799 9242 consensus_queue.cc:237] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66e293775084aada1d1b32ec78ca9fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42423 } }
I20260926 03:07:01.750082 9238 sys_catalog.cc:564] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:07:01.752337 9243 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b66e293775084aada1d1b32ec78ca9fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66e293775084aada1d1b32ec78ca9fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42423 } } }
I20260926 03:07:01.752748 9243 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [sys.catalog]: This master's current role is: LEADER
I20260926 03:07:01.753600 9244 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader b66e293775084aada1d1b32ec78ca9fa. Latest consensus state: current_term: 2 leader_uuid: "b66e293775084aada1d1b32ec78ca9fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66e293775084aada1d1b32ec78ca9fa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42423 } } }
I20260926 03:07:01.754047 9244 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa [sys.catalog]: This master's current role is: LEADER
I20260926 03:07:01.759562 9247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:07:01.772024 9247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:07:01.774173 9247 catalog_manager.cc:1259] Loaded cluster ID: bd18aa6295d941a5966590561c8e5f34
I20260926 03:07:01.774391 9247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:07:01.779079 9247 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:07:01.780937 9247 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b66e293775084aada1d1b32ec78ca9fa: Loaded TSK: 0
I20260926 03:07:01.783073 9247 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 03:07:02.051817 9240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:07:02.052119 9240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:07:02.052237 9240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:07:02.059980 9240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:07:02.060101 9240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:07:02.060245 9240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:07:02.071290 9240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:44599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38335
--tserver_master_addrs=127.0.97.62:42423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:07:02.072722 9240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:07:02.074290 9240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:07:02.094066 9264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:02.094098 9265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:02.094924 9267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:07:02.100585 9240 server_base.cc:1034] running on GCE node
I20260926 03:07:02.101609 9240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:07:02.103026 9240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:07:02.104313 9240 hybrid_clock.cc:648] HybridClock initialized: now 1790392022104237 us; error 69 us; skew 500 ppm
I20260926 03:07:02.104744 9240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:07:02 dist-test-slave-hmbw krb5kdc[8652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392022, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:07:02.109059 9240 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260926 03:07:02.110883 9240 webserver.cc:460] Webserver started at http://127.0.97.1:38335/ using document root <none> and password file <none>
I20260926 03:07:02.111660 9240 fs_manager.cc:359] Metadata directory not provided
I20260926 03:07:02.111794 9240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:07:02.118203 9240 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 03:07:02.121536 9274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:02.123345 9240 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:07:02.123550 9240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "d321135811ab4631b44a47f2df504e0e"
format_stamp: "Formatted at 2026-09-26 03:06:59 on dist-test-slave-hmbw"
server_key: "e743a3bdf8e68a12f349f2cf5fb763ea"
server_key_iv: "a2d997f6905e6a5833c8496f7beb1dfc"
server_key_version: "encryptionkey@0"
I20260926 03:07:02.125005 9240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:07:02.158820 9240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:07:02.163585 9240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:07:02.163826 9240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:07:02.164819 9240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:07:02.166198 9240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:07:02.166287 9240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:02.166363 9240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:07:02.166422 9240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:02.198051 9240 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44599
I20260926 03:07:02.198127 9386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44599 every 8 connection(s)
I20260926 03:07:02.199515 9240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:07:02.209992 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 9240
I20260926 03:07:02.210753 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:41937
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:42423
--webserver_port=38029
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:07:02 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:02 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:02.220054 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:02.202771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.1:48003)
0926 03:07:02.203621 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:02.203664 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:02.205454 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:02.206969 (+ 1515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:02.206981 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:02.207599 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:02.208127 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:02.208147 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:02.210327 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:02.210331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:02.211831 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:02.211859 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:02.213368 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:02.213375 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:02.213585 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:02.214326 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:02.214353 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:02.218435 (+ 4082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":311,"threads_started":1}
W20260926 03:07:02.220073 9389 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:02.202753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48003 (local address 127.0.97.62:42423)
0926 03:07:02.203526 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:02.203541 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:02.203776 (+ 235us) server_negotiation.cc:407] Connection header received
0926 03:07:02.206045 (+ 2269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:02.206057 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:02.206419 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:02.206822 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:02.208318 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:02.210133 (+ 1815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:02.212036 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:02.213066 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:02.213264 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:02.219001 (+ 5737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48003: BlockingRecv error: recv got EOF from 127.0.97.1:48003 (error 108)
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":292,"threads_started":1}
W20260926 03:07:02.221174 9387 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42423 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:42423: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 26 03:07:02 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:02 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:02.233824 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:02.222303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.1:57611)
0926 03:07:02.222957 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:02.222990 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:02.223260 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:02.223669 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:02.223675 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:02.224147 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:02.224341 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:02.224352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:02.226173 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:02.226180 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:02.227288 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:02.227311 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:02.229606 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:02.229614 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:02.229864 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:02.230637 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:02.230657 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:02.233524 (+ 2867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":83}
W20260926 03:07:02.234015 9389 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:02.222875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57611 (local address 127.0.97.62:42423)
0926 03:07:02.223140 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:02.223147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:02.223174 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:07:02.223427 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:02.223432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:02.223512 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:02.223611 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:02.224516 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:02.225579 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:02.227626 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:02.229382 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:02.229566 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:02.233795 (+ 4229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57611: BlockingRecv error: recv got EOF from 127.0.97.1:57611 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 26 03:07:02 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:02 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:02.247650 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:02.235401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.1:55825)
0926 03:07:02.235753 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:02.235774 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:02.235910 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:02.236367 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:02.236380 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:02.236742 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:02.236925 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:02.236934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:02.238125 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:02.238132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:02.239731 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:02.239749 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:02.241037 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:02.241047 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:02.241332 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:02.242563 (+ 1231us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:02.242596 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:02.247313 (+ 4717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213}
W20260926 03:07:02.247988 9387 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 03:07:02.248121 9389 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:02.235586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55825 (local address 127.0.97.62:42423)
0926 03:07:02.235777 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:02.235782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:02.235831 (+ 49us) server_negotiation.cc:407] Connection header received
0926 03:07:02.236037 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:02.236041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:02.236193 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:02.236307 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:02.237061 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:02.237970 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:02.239892 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:02.240462 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:02.240768 (+ 306us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:02.247607 (+ 6839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55825: BlockingRecv error: recv got EOF from 127.0.97.1:55825 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260926 03:07:02.616643 9391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:07:02.617019 9391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:07:02.617175 9391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:07:02.625082 9391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:07:02.625231 9391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:07:02.625381 9391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:07:02.636142 9391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:41937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=38029
--tserver_master_addrs=127.0.97.62:42423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:07:02.637671 9391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:07:02.639428 9391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:07:02.656291 9396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:02.656255 9399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:02.656345 9397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:07:02.657486 9391 server_base.cc:1034] running on GCE node
I20260926 03:07:02.658337 9391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:07:02.659531 9391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:07:02.660904 9391 hybrid_clock.cc:648] HybridClock initialized: now 1790392022660817 us; error 50 us; skew 500 ppm
I20260926 03:07:02.661947 9391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:07:02 dist-test-slave-hmbw krb5kdc[8652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392022, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:07:02.666678 9391 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260926 03:07:02.668672 9391 webserver.cc:460] Webserver started at http://127.0.97.2:38029/ using document root <none> and password file <none>
I20260926 03:07:02.669504 9391 fs_manager.cc:359] Metadata directory not provided
I20260926 03:07:02.669641 9391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:07:02.676254 9391 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 03:07:02.680099 9406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:02.681870 9391 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 03:07:02.682120 9391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "65ad33ce1ce94b50a878adc4d27a1cd8"
format_stamp: "Formatted at 2026-09-26 03:06:59 on dist-test-slave-hmbw"
server_key: "e2e58f0dad214ad82e32ce5d1dcc2f25"
server_key_iv: "9f36c4962b61b994d0da7501673d3ff8"
server_key_version: "encryptionkey@0"
I20260926 03:07:02.683344 9391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:07:02.697147 9391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:07:02.701767 9391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:07:02.702057 9391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:07:02.703168 9391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:07:02.704803 9391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:07:02.704914 9391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:02.705001 9391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:07:02.705072 9391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:02.745437 9391 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41937
I20260926 03:07:02.745445 9518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41937 every 8 connection(s)
I20260926 03:07:02.747116 9391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:07:02.750874 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 9391
I20260926 03:07:02.751531 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:41513
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:42423
--webserver_port=33577
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:07:02 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:02 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:07:02.771844 9522 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:02.753180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39477 (local address 127.0.97.62:42423)
0926 03:07:02.756978 (+ 3798us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:02.756985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:02.757007 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:07:02.757068 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:02.757073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:02.757169 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:02.757288 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:02.759654 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:02.760718 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:02.763392 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:02.764026 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:02.764195 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:02.771456 (+ 7261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39477: BlockingRecv error: recv got EOF from 127.0.97.2:39477 (error 108)
Metrics: {"server-negotiator.queue_time_us":3650,"thread_start_us":3483,"threads_started":1}
I20260926 03:07:02.772256 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:02.752398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.2:39477)
0926 03:07:02.753607 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:02.753680 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:02.756101 (+ 2421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:02.757667 (+ 1566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:02.757681 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:02.758450 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:02.759358 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:02.759407 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:02.761080 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:02.761088 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:02.763119 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:02.763150 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:02.764421 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:02.764429 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:02.764835 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:02.765811 (+ 976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:02.765849 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:02.770971 (+ 5122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":784,"thread_start_us":469,"threads_started":1}
W20260926 03:07:02.773231 9519 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42423 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:42423: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 26 03:07:02 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:02 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:02.786984 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:02.774147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.2:60525)
0926 03:07:02.774491 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:02.774544 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:02.774672 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:02.775285 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:02.775289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:02.775799 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:02.775979 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:02.775989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:02.777858 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:02.777863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:02.779236 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:02.779254 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:02.780457 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:02.780462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:02.780936 (+ 474us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:02.781794 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:02.781820 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:02.786759 (+ 4939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":74}
W20260926 03:07:02.787405 9522 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:02.774641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60525 (local address 127.0.97.62:42423)
0926 03:07:02.774882 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:02.774889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:02.774915 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:07:02.774988 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:02.774993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:02.775084 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:02.775211 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:02.776155 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:02.777541 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:02.779420 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:02.780212 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:02.780398 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:02.787079 (+ 6681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:60525: BlockingRecv error: recv got EOF from 127.0.97.2:60525 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 26 03:07:02 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:02 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:02.801271 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:02.787740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.2:33911)
0926 03:07:02.787937 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:02.787960 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:02.788237 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:02.789158 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:02.789164 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:02.789740 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:02.790015 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:02.790030 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:02.792151 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:02.792158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:02.794052 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:02.794079 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:02.795628 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:02.795637 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:02.795876 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:02.796702 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:02.796730 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:02.800937 (+ 4207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":10760,"threads_started":1}
W20260926 03:07:02.801445 9522 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:02.787813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33911 (local address 127.0.97.62:42423)
0926 03:07:02.788588 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:02.788595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:02.788617 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:07:02.788689 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:02.788694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:02.788811 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:02.788939 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:02.790270 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:02.791896 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:02.794350 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:02.795164 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:02.795308 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:02.801208 (+ 5900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33911: BlockingRecv error: recv got EOF from 127.0.97.2:33911 (error 108)
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":8700,"threads_started":1}
W20260926 03:07:02.801643 9519 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 03:07:03.127151 9523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:07:03.127543 9523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:07:03.127697 9523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:07:03.135914 9523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:07:03.136041 9523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:07:03.136154 9523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:07:03.146180 9523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:41513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=33577
--tserver_master_addrs=127.0.97.62:42423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:07:03.147574 9523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:07:03.148873 9523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:07:03.162267 9533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:03.162289 9530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:03.162276 9531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:07:03.162899 9523 server_base.cc:1034] running on GCE node
I20260926 03:07:03.163543 9523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:07:03.164461 9523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:07:03.165664 9523 hybrid_clock.cc:648] HybridClock initialized: now 1790392023165611 us; error 51 us; skew 500 ppm
I20260926 03:07:03.165957 9523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:07:03 dist-test-slave-hmbw krb5kdc[8652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392023, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:07:03.169096 9523 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260926 03:07:03.170589 9523 webserver.cc:460] Webserver started at http://127.0.97.3:33577/ using document root <none> and password file <none>
I20260926 03:07:03.171271 9523 fs_manager.cc:359] Metadata directory not provided
I20260926 03:07:03.171392 9523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:07:03.176898 9523 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 03:07:03.179661 9540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:03.181255 9523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:07:03.181429 9523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "9ea4e0742ee84d818ef25c759ae54990"
format_stamp: "Formatted at 2026-09-26 03:07:00 on dist-test-slave-hmbw"
server_key: "287e089aab5309877883063f215345b1"
server_key_iv: "503a2083fe149441191255a58435be1f"
server_key_version: "encryptionkey@0"
I20260926 03:07:03.182358 9523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:07:03.201541 9523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:07:03.205583 9523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:07:03.205824 9523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:07:03.206881 9523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:07:03.208259 9523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:07:03.208351 9523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:03.208453 9523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:07:03.208526 9523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:03.238770 9523 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41513
I20260926 03:07:03.238817 9652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41513 every 8 connection(s)
I20260926 03:07:03.240163 9523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:07:03.248814 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 9523
Sep 26 03:07:03 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:03 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:03 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:07:03.258080 9522 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:03.242641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47493 (local address 127.0.97.62:42423)
0926 03:07:03.242829 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:03.242834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:03.244109 (+ 1275us) server_negotiation.cc:407] Connection header received
0926 03:07:03.245951 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:03.245955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:03.246091 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:03.246191 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:03.247688 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:03.248553 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:03.251080 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:03.251788 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:03.251893 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:03.257809 (+ 5916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47493: BlockingRecv error: recv got EOF from 127.0.97.3:47493 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:07:03.258996 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:03.243234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.3:47493)
0926 03:07:03.243943 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:03.243987 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:03.245675 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:03.246402 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:03.246410 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:03.246937 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:03.247502 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:03.247519 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:03.248722 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:03.248727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:03.250913 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:03.250928 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:03.252218 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:03.252225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:03.252511 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:03.253346 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:03.253386 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:03.257612 (+ 4226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":276,"threads_started":1}
W20260926 03:07:03.259758 9653 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42423 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:42423: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 26 03:07:03 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:03.261122 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:03.248530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.1:37819)
0926 03:07:03.249547 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:03.249565 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:03.249723 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:03.250285 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:03.250289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:03.250909 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:03.251141 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:03.251154 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:03.252800 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:03.252805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:03.254391 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:03.254409 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:03.255480 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:03.255487 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:03.255658 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:03.256261 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:03.256285 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:03.260831 (+ 4546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":889,"thread_start_us":831,"threads_started":1}
W20260926 03:07:03.261380 9525 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:03.248793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37819 (local address 127.0.97.62:42423)
0926 03:07:03.249082 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:03.249087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:03.249633 (+ 546us) server_negotiation.cc:407] Connection header received
0926 03:07:03.249859 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:03.249865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:03.249935 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:03.250011 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:03.251305 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:03.252584 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:03.254592 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:03.255291 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:03.255416 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:03.261099 (+ 5683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37819: BlockingRecv error: recv got EOF from 127.0.97.1:37819 (error 108)
Metrics: {"server-negotiator.queue_time_us":153}
Sep 26 03:07:03 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:03 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:07:03.268652 9522 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:03.260508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44751 (local address 127.0.97.62:42423)
0926 03:07:03.260740 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:03.260746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:03.260763 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:07:03.260940 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:03.260946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:03.261027 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:03.261126 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:03.262019 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:03.262875 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:03.264033 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:03.264511 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:03.264630 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:03.268390 (+ 3760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44751: BlockingRecv error: recv got EOF from 127.0.97.3:44751 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
I20260926 03:07:03.269049 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:03.260400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.3:44751)
0926 03:07:03.260630 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:03.260647 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:03.260742 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:03.261330 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:03.261335 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:03.261718 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:03.261883 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:03.261892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:03.263000 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:03.263005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:03.263898 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:03.263910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:03.264657 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:03.264662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:03.264820 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:03.265324 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:03.265348 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:03.268862 (+ 3514us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132}
I20260926 03:07:03.271684 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:03.249707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33074 (local address 127.0.97.62:42423)
0926 03:07:03.255296 (+ 5589us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:03.255302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:03.255905 (+ 603us) server_negotiation.cc:407] Connection header received
0926 03:07:03.255961 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:03.255965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:03.256034 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:03.256130 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:03.258304 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:03.259442 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:03.261226 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:03.261813 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:03.261919 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:03.264404 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:03.264412 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:03.264432 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:07:03.264474 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:03.267976 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:07:03.268810 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:07:03.268818 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:07:03.268828 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:07:03.269100 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:07:03.269597 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:07:03.269602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:07:03.269605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:07:03.270185 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:07:03.270542 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:03.271021 (+ 479us) server_negotiation.cc:299] Negotiation successful
0926 03:07:03.271394 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":5477,"thread_start_us":5325,"threads_started":1}
Sep 26 03:07:03 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:03 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:03.278048 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:03.269745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.3:60895)
0926 03:07:03.269906 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:03.269925 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:03.270020 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:03.270388 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:03.270393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:03.270716 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:03.270879 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:03.270888 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:03.272004 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:03.272010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:03.273238 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:03.273250 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:03.274189 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:03.274195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:03.274360 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:03.274977 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:03.274999 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:03.277762 (+ 2763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 03:07:03.278162 9522 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:03.269821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60895 (local address 127.0.97.62:42423)
0926 03:07:03.270015 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:03.270020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:03.270036 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:07:03.270125 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:03.270134 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:03.270212 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:03.270285 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:03.271006 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:03.271863 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:03.273421 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:03.274042 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:03.274157 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:03.277959 (+ 3802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60895: BlockingRecv error: recv got EOF from 127.0.97.3:60895 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
W20260926 03:07:03.278362 9653 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 03:07:03 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:03 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:03.811429 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:03.802355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.2:50537)
0926 03:07:03.802831 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:03.802847 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:03.802949 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:03.803297 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:03.803301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:03.803739 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:03.803958 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:03.803969 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:03.805087 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:03.805091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:03.806098 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:03.806112 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:03.806904 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:03.806910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:03.807052 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:03.807665 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:03.807716 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:03.811121 (+ 3405us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":252,"threads_started":1}
W20260926 03:07:03.811803 9660 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:03.802440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50537 (local address 127.0.97.62:42423)
0926 03:07:03.802888 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:03.802893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:03.802911 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:07:03.803047 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:03.803051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:03.803111 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:03.803206 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:03.804146 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:03.804928 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:03.806252 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:03.806716 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:03.806825 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:03.811543 (+ 4718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50537: BlockingRecv error: recv got EOF from 127.0.97.2:50537 (error 108)
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":261,"threads_started":1}
Sep 26 03:07:04 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:04 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:04.270912 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:04.262320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.1:37199)
0926 03:07:04.262817 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:04.262833 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:04.262954 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:04.263261 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:04.263264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:04.263687 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:04.263830 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:04.263838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:04.265005 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:04.265009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:04.265934 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:04.265947 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:04.266868 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:04.266875 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:04.267043 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:04.267727 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:04.267746 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:04.270672 (+ 2926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":277,"threads_started":1}
W20260926 03:07:04.271179 9660 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:04.262418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37199 (local address 127.0.97.62:42423)
0926 03:07:04.262611 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:04.262615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:04.262939 (+ 324us) server_negotiation.cc:407] Connection header received
0926 03:07:04.263065 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:04.263069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:04.263134 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:04.263223 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:04.263990 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:04.264795 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:04.266142 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:04.266698 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:04.266832 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:04.270901 (+ 4069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37199: BlockingRecv error: recv got EOF from 127.0.97.1:37199 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 03:07:04 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:04 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:04.287705 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:04.278933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.3:51551)
0926 03:07:04.279540 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:04.279561 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:04.279701 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:04.280091 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:04.280095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:04.280577 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:04.280844 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:04.280855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:04.281994 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:04.281999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:04.283125 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:04.283141 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:04.284029 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:04.284035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:04.284152 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:04.284716 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:04.284736 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:04.287424 (+ 2688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":335,"threads_started":1}
W20260926 03:07:04.287814 9660 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:04.279015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51551 (local address 127.0.97.62:42423)
0926 03:07:04.279202 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:04.279209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:04.279652 (+ 443us) server_negotiation.cc:407] Connection header received
0926 03:07:04.279825 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:04.279830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:04.279911 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:04.280055 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:04.281070 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:04.281808 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:04.283291 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:04.283879 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:04.284028 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:04.287606 (+ 3578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51551: BlockingRecv error: recv got EOF from 127.0.97.3:51551 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 03:07:04 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:04 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:04.821679 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:04.813155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.2:35239)
0926 03:07:04.813655 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:04.813673 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:04.813795 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:04.814268 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:04.814271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:04.814686 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:04.814829 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:04.814838 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:04.815926 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:04.815931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:04.816827 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:04.816839 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:04.817576 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:04.817582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:04.817704 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:04.818297 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:04.818315 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:04.821389 (+ 3074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":285,"threads_started":1}
W20260926 03:07:04.822125 9664 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:04.813300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35239 (local address 127.0.97.62:42423)
0926 03:07:04.813953 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:04.813957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:04.813973 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:07:04.814025 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:04.814028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:04.814115 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:04.814201 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:04.814991 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:04.815768 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:04.816970 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:04.817412 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:04.817511 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:04.821662 (+ 4151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35239: BlockingRecv error: recv got EOF from 127.0.97.2:35239 (error 108)
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":435,"threads_started":1}
Sep 26 03:07:05 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:05 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:07:05.288734 9664 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:05.272154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49477 (local address 127.0.97.62:42423)
0926 03:07:05.272366 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:05.272372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:05.272593 (+ 221us) server_negotiation.cc:407] Connection header received
0926 03:07:05.272725 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:05.272729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:05.272792 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:05.272875 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:05.273586 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:05.274336 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:05.275814 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:05.276389 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:05.276492 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:05.288489 (+ 11997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49477: BlockingRecv error: recv got EOF from 127.0.97.1:49477 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:07:05.288735 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:05.272046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.1:49477)
0926 03:07:05.272492 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:05.272510 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:05.272631 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:05.272908 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:05.272912 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:05.273296 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:05.273465 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:05.273476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:05.274476 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:05.274482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:05.275633 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:05.275648 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:05.276525 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:05.276529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:05.276659 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:05.284963 (+ 8304us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:05.284990 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:05.288264 (+ 3274us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":242,"threads_started":1}
Sep 26 03:07:05 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:05 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:05.296622 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:05.288789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.3:44873)
0926 03:07:05.289366 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:05.289390 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:05.289508 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:05.289792 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:05.289795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:05.290197 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:05.290344 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:05.290353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:05.291437 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:05.291442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:05.292339 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:05.292352 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:05.293052 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:05.293057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:05.293192 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:05.293711 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:05.293727 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:05.296354 (+ 2627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":353,"threads_started":1}
W20260926 03:07:05.296803 9664 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:05.289087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44873 (local address 127.0.97.62:42423)
0926 03:07:05.289258 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:05.289262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:05.289484 (+ 222us) server_negotiation.cc:407] Connection header received
0926 03:07:05.289602 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:05.289606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:05.289664 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:05.289745 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:05.290517 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:05.291281 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:05.292472 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:05.292883 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:05.293040 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:05.296562 (+ 3522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44873: BlockingRecv error: recv got EOF from 127.0.97.3:44873 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 03:07:05 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:05 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:05.831301 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:05.822779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.2:44473)
0926 03:07:05.823251 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:05.823272 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:05.823384 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:05.823684 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:05.823687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:05.824062 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:05.824235 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:05.824244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:05.825292 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:05.825297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:05.826236 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:05.826248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:05.827040 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:05.827046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:05.827203 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:05.827856 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:05.827874 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:05.830966 (+ 3092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":262,"threads_started":1}
W20260926 03:07:05.831442 9668 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:05.822944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44473 (local address 127.0.97.62:42423)
0926 03:07:05.823381 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:05.823385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:05.823403 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:07:05.823460 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:05.823464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:05.823523 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:05.823627 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:05.824395 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:05.825137 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:05.826372 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:05.826857 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:05.826976 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:05.831201 (+ 4225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44473: BlockingRecv error: recv got EOF from 127.0.97.2:44473 (error 108)
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":248,"threads_started":1}
Sep 26 03:07:06 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:06 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:06.297928 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:06.289621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.1:42725)
0926 03:07:06.290157 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:06.290172 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:06.290301 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:06.290640 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:06.290645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:06.291008 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:06.291157 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:06.291165 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:06.292134 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:06.292140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:06.293067 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:06.293079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:06.293782 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:06.293787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:06.294008 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:06.294686 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:06.294704 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:06.297620 (+ 2916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":267,"threads_started":1}
W20260926 03:07:06.298286 9668 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:06.289730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42725 (local address 127.0.97.62:42423)
0926 03:07:06.289983 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:06.289990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:06.290254 (+ 264us) server_negotiation.cc:407] Connection header received
0926 03:07:06.290421 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:06.290426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:06.290517 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:06.290599 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:06.291297 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:06.292000 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:06.293206 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:06.293611 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:06.293733 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:06.297845 (+ 4112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42725: BlockingRecv error: recv got EOF from 127.0.97.1:42725 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 03:07:06 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:06 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:06.308211 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:06.297632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.3:49917)
0926 03:07:06.299446 (+ 1814us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:06.299462 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:06.299596 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:06.299992 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:06.299996 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:06.300308 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:06.300481 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:06.300493 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:06.301738 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:06.301742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:06.302757 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:06.302772 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:06.303658 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:06.303665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:06.303849 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:06.304577 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:06.304598 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:06.307758 (+ 3160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1721,"thread_start_us":1619,"threads_started":1}
W20260926 03:07:06.308282 9668 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:06.297745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49917 (local address 127.0.97.62:42423)
0926 03:07:06.298990 (+ 1245us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:06.298997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:06.299590 (+ 593us) server_negotiation.cc:407] Connection header received
0926 03:07:06.299696 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:06.299700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:06.299778 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:06.299865 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:06.300645 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:06.301588 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:06.302920 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:06.303482 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:06.303616 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:06.308024 (+ 4408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49917: BlockingRecv error: recv got EOF from 127.0.97.3:49917 (error 108)
Metrics: {"server-negotiator.queue_time_us":1059,"thread_start_us":1426,"threads_started":1}
Sep 26 03:07:06 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:06 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:06.840948 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:06.832274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.2:52773)
0926 03:07:06.832718 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:06.832735 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:06.832842 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:06.833277 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:06.833280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:06.833675 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:06.833855 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:06.833865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:06.834987 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:06.834992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:06.835948 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:06.835959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:06.836673 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:06.836678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:06.836840 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:06.837453 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:06.837477 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:06.840610 (+ 3133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":234,"threads_started":1}
W20260926 03:07:06.841070 9673 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:06.832362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52773 (local address 127.0.97.62:42423)
0926 03:07:06.832958 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:06.832963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:06.832983 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:07:06.833035 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:06.833038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:06.833107 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:06.833205 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:06.834014 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:06.834787 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:06.836087 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:06.836514 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:06.836611 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:06.840833 (+ 4222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52773: BlockingRecv error: recv got EOF from 127.0.97.2:52773 (error 108)
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":255,"threads_started":1}
Sep 26 03:07:07 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:07 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:07.307691 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:07.299257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.1:44957)
0926 03:07:07.299799 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:07.299814 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:07.299944 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:07.300282 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:07.300286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:07.300729 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:07.300895 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:07.300904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:07.301905 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:07.301909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:07.302994 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:07.303010 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:07.303723 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:07.303730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:07.303890 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:07.304473 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:07.304491 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:07.307441 (+ 2950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":287,"threads_started":1}
W20260926 03:07:07.308074 9673 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:07.299397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44957 (local address 127.0.97.62:42423)
0926 03:07:07.299641 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:07.299648 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:07.299898 (+ 250us) server_negotiation.cc:407] Connection header received
0926 03:07:07.300061 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:07.300067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:07.300144 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:07.300231 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:07.301043 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:07.301765 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:07.303148 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:07.303558 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:07.303680 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:07.307756 (+ 4076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:44957: BlockingRecv error: recv got EOF from 127.0.97.1:44957 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 26 03:07:07 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:07 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:07.317636 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:07.309317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.3:59593)
0926 03:07:07.309970 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:07.310028 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:07.310145 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:07.310415 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:07.310418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:07.310815 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:07.310962 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:07.310972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:07.312096 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:07.312100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:07.313036 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:07.313048 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:07.313715 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:07.313718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:07.313847 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:07.314453 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:07.314471 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:07.317384 (+ 2913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":414,"threads_started":1}
W20260926 03:07:07.317902 9673 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:07.309452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59593 (local address 127.0.97.62:42423)
0926 03:07:07.309621 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:07.309625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:07.310140 (+ 515us) server_negotiation.cc:407] Connection header received
0926 03:07:07.310231 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:07.310236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:07.310300 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:07.310384 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:07.311151 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:07.311962 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:07.313186 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:07.313596 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:07.313696 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:07.317641 (+ 3945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59593: BlockingRecv error: recv got EOF from 127.0.97.3:59593 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 03:07:07 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:07 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:07.850855 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:07.841980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.2:41721)
0926 03:07:07.842456 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:07.842476 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:07.842621 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:07.843020 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:07.843023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:07.843385 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:07.843557 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:07.843566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:07.844731 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:07.844735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:07.845660 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:07.845672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:07.846567 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:07.846574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:07.846721 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:07.847292 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:07.847310 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:07.850559 (+ 3249us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":225,"threads_started":1}
W20260926 03:07:07.851037 9677 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:07.842225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41721 (local address 127.0.97.62:42423)
0926 03:07:07.842717 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:07.842722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:07.842738 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:07:07.842790 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:07.842793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:07.842880 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:07.842971 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:07.843722 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:07.844565 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:07.845821 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:07.846327 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:07.846472 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:07.850803 (+ 4331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41721: BlockingRecv error: recv got EOF from 127.0.97.2:41721 (error 108)
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":315,"threads_started":1}
Sep 26 03:07:08 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:08 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:07:08.319315 9677 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:08.308956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42909 (local address 127.0.97.62:42423)
0926 03:07:08.309295 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:08.309304 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:08.309721 (+ 417us) server_negotiation.cc:407] Connection header received
0926 03:07:08.309940 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:08.309947 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:08.310020 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:08.310112 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:08.311174 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:08.311947 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:08.313261 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:08.313697 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:08.313818 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:08.319078 (+ 5260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42909: BlockingRecv error: recv got EOF from 127.0.97.1:42909 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
I20260926 03:07:08.319706 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:08.308813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.1:42909)
0926 03:07:08.309572 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:08.309592 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:08.309796 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:08.310262 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:08.310269 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:08.310774 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:08.310957 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:08.310967 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:08.312122 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:08.312126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:08.313102 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:08.313116 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:08.313871 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:08.313877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:08.314041 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:08.314797 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:08.314875 (+ 78us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:08.319411 (+ 4536us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":486,"threads_started":1}
Sep 26 03:07:08 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:08 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:08.328011 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:08.318717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.3:47111)
0926 03:07:08.319464 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:08.319488 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:08.319609 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:08.320197 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:08.320202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:08.320680 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:08.320844 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:08.320854 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:08.321993 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:08.321998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:08.323061 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:08.323076 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:08.323826 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:08.323831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:08.324031 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:08.324656 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:08.324675 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:08.327730 (+ 3055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":417,"threads_started":1}
W20260926 03:07:08.328312 9677 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:08.318912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47111 (local address 127.0.97.62:42423)
0926 03:07:08.319575 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:08.319582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:08.319604 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:07:08.319765 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:08.319770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:08.319879 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:08.319969 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:08.320986 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:08.321778 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:08.323224 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:08.323660 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:08.323768 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:08.327980 (+ 4212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47111: BlockingRecv error: recv got EOF from 127.0.97.3:47111 (error 108)
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":1254,"threads_started":1}
Sep 26 03:07:08 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:08 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:08.861325 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:08.851948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.2:56811)
0926 03:07:08.852557 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:08.852579 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:08.852711 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:08.853026 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:08.853029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:08.853375 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:08.853521 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:08.853531 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:08.854780 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:08.854785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:08.855942 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:08.855969 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:08.856928 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:08.856934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:08.857077 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:08.857780 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:08.857810 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:08.860977 (+ 3167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":276,"threads_started":1}
W20260926 03:07:08.861641 9682 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:08.852024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56811 (local address 127.0.97.62:42423)
0926 03:07:08.852717 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:08.852721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:08.852738 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:07:08.852792 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:08.852795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:08.852853 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:08.852942 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:08.853690 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:08.854566 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:08.856133 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:08.856698 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:08.856836 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:08.861266 (+ 4430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56811: BlockingRecv error: recv got EOF from 127.0.97.2:56811 (error 108)
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":525,"threads_started":1}
Sep 26 03:07:09 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:09 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:09.330173 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:09.320767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.1:45267)
0926 03:07:09.321422 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:09.321437 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:09.321585 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:09.321896 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:09.321901 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:09.322342 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:09.322603 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:09.322615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:09.323639 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:09.323644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:09.324608 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:09.324620 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:09.325301 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:09.325307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:09.325466 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:09.326147 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:09.326191 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:09.329800 (+ 3609us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":281,"threads_started":1}
W20260926 03:07:09.330257 9682 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:09.321024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45267 (local address 127.0.97.62:42423)
0926 03:07:09.321267 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:09.321273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:09.321540 (+ 267us) server_negotiation.cc:407] Connection header received
0926 03:07:09.321710 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:09.321715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:09.321774 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:09.321865 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:09.322763 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:09.323504 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:09.324746 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:09.325150 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:09.325270 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:09.330028 (+ 4758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45267: BlockingRecv error: recv got EOF from 127.0.97.1:45267 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 26 03:07:09 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:09 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:09.337707 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:09.329071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.3:54355)
0926 03:07:09.329710 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:09.329732 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:09.329884 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:09.330842 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:09.330848 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:09.331182 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:09.331332 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:09.331341 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:09.332366 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:09.332370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:09.333267 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:09.333280 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:09.333983 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:09.333989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:09.334155 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:09.334715 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:09.334732 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:09.337404 (+ 2672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":289,"threads_started":1}
W20260926 03:07:09.337992 9684 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:09.329087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54355 (local address 127.0.97.62:42423)
0926 03:07:09.330393 (+ 1306us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:09.330398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:09.330415 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:07:09.330465 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:09.330476 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:09.330572 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:09.330668 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:09.331480 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:09.332232 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:09.333403 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:09.333826 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:09.333929 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:09.337703 (+ 3774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:54355: BlockingRecv error: recv got EOF from 127.0.97.3:54355 (error 108)
Metrics: {"server-negotiator.queue_time_us":1189,"thread_start_us":1000,"threads_started":1}
Sep 26 03:07:09 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:09 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:09.871095 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:09.862408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.2:37979)
0926 03:07:09.863162 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:09.863186 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:09.863341 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:09.863702 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:09.863706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:09.864119 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:09.864277 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:09.864286 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:09.865254 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:09.865258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:09.866165 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:09.866177 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:09.866921 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:09.866927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:09.867086 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:09.867654 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:09.867672 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:09.870790 (+ 3118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":299,"threads_started":1}
W20260926 03:07:09.871454 9687 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:09.862582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37979 (local address 127.0.97.62:42423)
0926 03:07:09.863340 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:09.863344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:09.863364 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:07:09.863465 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:09.863468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:09.863544 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:09.863641 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:09.864416 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:09.865114 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:09.866301 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:09.866775 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:09.866884 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:09.871005 (+ 4121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37979: BlockingRecv error: recv got EOF from 127.0.97.2:37979 (error 108)
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":572,"threads_started":1}
Sep 26 03:07:10 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:10 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:10 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:10 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:10.349952 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:10.338777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.3:40607)
0926 03:07:10.339374 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:10.339391 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:10.339547 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:10.339854 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:10.339857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:10.340254 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:10.340462 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:10.340474 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:10.341653 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:10.341658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:10.342649 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:10.342663 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:10.343468 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:10.343474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:10.343596 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:10.344912 (+ 1316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:10.344986 (+ 74us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:10.349596 (+ 4610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":246,"threads_started":1}
I20260926 03:07:10.350354 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:10.331602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.1:40393)
0926 03:07:10.332126 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:10.332142 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:10.332262 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:10.332609 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:10.332613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:10.332989 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:10.333178 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:10.333186 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:10.334287 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:10.334291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:10.335285 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:10.335297 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:10.336061 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:10.336068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:10.336198 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:10.344916 (+ 8718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:10.344940 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:10.350046 (+ 5106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":279,"threads_started":1}
W20260926 03:07:10.350458 9687 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:10.331730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40393 (local address 127.0.97.62:42423)
0926 03:07:10.331892 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:10.331896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:10.332245 (+ 349us) server_negotiation.cc:407] Connection header received
0926 03:07:10.332366 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:10.332370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:10.332444 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:10.332547 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:10.333341 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:10.334132 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:10.335437 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:10.335853 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:10.336046 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:10.350224 (+ 14178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:40393: BlockingRecv error: recv got EOF from 127.0.97.1:40393 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 03:07:10.350893 9690 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:10.338974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40607 (local address 127.0.97.62:42423)
0926 03:07:10.339610 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:10.339614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:10.339628 (+ 14us) server_negotiation.cc:407] Connection header received
0926 03:07:10.339672 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:10.339675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:10.339731 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:10.339818 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:10.340620 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:10.341482 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:10.342788 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:10.343266 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:10.343363 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:10.349865 (+ 6502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40607: BlockingRecv error: recv got EOF from 127.0.97.3:40607 (error 108)
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":430,"threads_started":1}
Sep 26 03:07:10 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:10 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:10.880712 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:10.872157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.2:56245)
0926 03:07:10.872654 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:10.872671 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:10.872778 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:10.873133 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:10.873136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:10.873543 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:10.873686 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:10.873694 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:10.874973 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:10.874978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:10.876041 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:10.876057 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:10.876917 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:10.876924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:10.877055 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:10.877672 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:10.877688 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:10.880477 (+ 2789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":237,"threads_started":1}
W20260926 03:07:10.880967 9692 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:10.872313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56245 (local address 127.0.97.62:42423)
0926 03:07:10.872771 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:10.872775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:10.872794 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:07:10.872878 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:10.872883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:10.872949 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:10.873053 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:10.873837 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:10.874815 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:10.876186 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:10.876671 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:10.876778 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:10.880705 (+ 3927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56245: BlockingRecv error: recv got EOF from 127.0.97.2:56245 (error 108)
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":264,"threads_started":1}
Sep 26 03:07:11 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:11 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:11 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:11 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:11.360773 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:11.351445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.1:38185)
0926 03:07:11.352207 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:11.352231 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:11.352500 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:11.353022 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:11.353025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:11.353467 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:11.353636 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:11.353645 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:11.354855 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:11.354861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:11.355797 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:11.355809 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:11.356509 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:11.356514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:11.356681 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:11.357271 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:11.357293 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:11.360507 (+ 3214us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":405,"threads_started":1}
W20260926 03:07:11.361084 9692 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:11.351425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38185 (local address 127.0.97.62:42423)
0926 03:07:11.351707 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:11.351712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:11.352357 (+ 645us) server_negotiation.cc:407] Connection header received
0926 03:07:11.352658 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:11.352665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:11.352753 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:11.352827 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:11.353779 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:11.354720 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:11.355936 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:11.356355 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:11.356465 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:11.360808 (+ 4343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38185: BlockingRecv error: recv got EOF from 127.0.97.1:38185 (error 108)
Metrics: {"server-negotiator.queue_time_us":164}
I20260926 03:07:11.361083 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:11.351340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.3:41071)
0926 03:07:11.352471 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:11.352490 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:11.352608 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:11.353359 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:11.353365 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:11.353721 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:11.353904 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:11.353918 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:11.355117 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:11.355123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:11.356044 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:11.356058 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:11.356898 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:11.356906 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:11.357064 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:11.357573 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:11.357590 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:11.360776 (+ 3186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":990,"thread_start_us":832,"threads_started":1}
W20260926 03:07:11.361342 9695 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:11.351470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41071 (local address 127.0.97.62:42423)
0926 03:07:11.353066 (+ 1596us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:11.353071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:11.353086 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:07:11.353128 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:11.353132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:11.353201 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:11.353291 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:11.354056 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:11.354976 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:11.356206 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:11.356694 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:11.356802 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:11.361098 (+ 4296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41071: BlockingRecv error: recv got EOF from 127.0.97.3:41071 (error 108)
Metrics: {"server-negotiator.queue_time_us":1500,"thread_start_us":1360,"threads_started":1}
Sep 26 03:07:11 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:11 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:11.890122 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:11.881684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.2:54921)
0926 03:07:11.882220 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:11.882235 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:11.882343 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:11.882673 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:11.882676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:11.883055 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:11.883216 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:11.883224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:11.884329 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:11.884334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:11.885477 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:11.885492 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:11.886355 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:11.886360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:11.886537 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:11.887146 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:11.887165 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:11.889830 (+ 2665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":330,"threads_started":1}
W20260926 03:07:11.890264 9697 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:11.881767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54921 (local address 127.0.97.62:42423)
0926 03:07:11.882240 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:11.882244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:11.882287 (+ 43us) server_negotiation.cc:407] Connection header received
0926 03:07:11.882464 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:11.882467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:11.882547 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:11.882639 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:11.883346 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:11.884175 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:11.885654 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:11.886183 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:11.886287 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:11.890044 (+ 3757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:54921: BlockingRecv error: recv got EOF from 127.0.97.2:54921 (error 108)
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":283,"threads_started":1}
Sep 26 03:07:12 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:12 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:12 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:12 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:12.372433 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:12.361992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.1:35795)
0926 03:07:12.362915 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:12.362938 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:12.363586 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:12.364247 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:12.364251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:12.364629 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:12.364771 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:12.364779 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:12.365937 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:12.365941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:12.367183 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:12.367200 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:12.368121 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:12.368137 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:12.368305 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:12.369025 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:12.369053 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:12.372082 (+ 3029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":789,"thread_start_us":667,"threads_started":1}
W20260926 03:07:12.372593 9697 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:12.362199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35795 (local address 127.0.97.62:42423)
0926 03:07:12.362646 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:12.362653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:12.363460 (+ 807us) server_negotiation.cc:407] Connection header received
0926 03:07:12.363916 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:12.363921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:12.363980 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:12.364030 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:12.364914 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:12.365782 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:12.367364 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:12.367949 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:12.368110 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:12.372328 (+ 4218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35795: BlockingRecv error: recv got EOF from 127.0.97.1:35795 (error 108)
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":313}
I20260926 03:07:12.372892 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:12.361964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.3:44309)
0926 03:07:12.363587 (+ 1623us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:12.363605 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:12.363716 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:12.364145 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:12.364150 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:12.364599 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:12.364771 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:12.364780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:12.365919 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:12.366002 (+ 83us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:12.367193 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:12.367210 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:12.368125 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:12.368132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:12.368317 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:12.369178 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:12.369196 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:12.372553 (+ 3357us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1309,"thread_start_us":1169,"threads_started":1}
W20260926 03:07:12.373111 9700 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:12.362348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44309 (local address 127.0.97.62:42423)
0926 03:07:12.363664 (+ 1316us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:12.363670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:12.363684 (+ 14us) server_negotiation.cc:407] Connection header received
0926 03:07:12.363839 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:12.363844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:12.363917 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:12.364019 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:12.364935 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:12.365773 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:12.367369 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:12.367961 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:12.368093 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:12.372806 (+ 4713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44309: BlockingRecv error: recv got EOF from 127.0.97.3:44309 (error 108)
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":1203,"thread_start_us":1075,"threads_started":1}
Sep 26 03:07:12 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:12 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:12.899264 9701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:12.891048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.2:36317)
0926 03:07:12.891503 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:12.891519 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:12.891612 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:12.891954 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:12.891957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:12.892277 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:12.892422 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:12.892430 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:12.893505 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:12.893509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:12.894488 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:12.894532 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:12.895278 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:12.895283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:12.895400 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:12.896049 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:12.896065 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:12.899033 (+ 2968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":255,"threads_started":1}
W20260926 03:07:12.899518 9702 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:12.891116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36317 (local address 127.0.97.62:42423)
0926 03:07:12.891541 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:12.891546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:12.891564 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:07:12.891736 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:12.891740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:12.891811 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:12.891930 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:12.892562 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:12.893347 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:12.894677 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:12.895108 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:12.895210 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:12.899240 (+ 4030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36317: BlockingRecv error: recv got EOF from 127.0.97.2:36317 (error 108)
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":230,"threads_started":1}
Sep 26 03:07:13 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:13 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:13 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:13 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:13.382434 9703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:13.373682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.1:35405)
0926 03:07:13.374340 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:13.374372 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:13.374485 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:13.374887 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:13.374891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:13.375361 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:13.375532 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:13.375541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:13.376879 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:13.376884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:13.377795 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:13.377807 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:13.378529 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:13.378536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:13.378709 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:13.379289 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:13.379313 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:13.382150 (+ 2837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":442,"threads_started":1}
I20260926 03:07:13.382648 9704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:13.373879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.3:48145)
0926 03:07:13.374363 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:13.374379 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:13.374484 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:13.375298 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:13.375303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:13.375641 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:13.375799 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:13.375811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:13.377056 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:13.377060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:13.377976 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:13.377987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:13.378907 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:13.378912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:13.379063 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:13.379497 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:13.379512 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:13.382326 (+ 2814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":273,"threads_started":1}
W20260926 03:07:13.382659 9702 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:13.374394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35405 (local address 127.0.97.62:42423)
0926 03:07:13.374611 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:13.374617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:13.374632 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:07:13.374679 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:13.374682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:13.374740 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:13.374830 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:13.375670 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:13.376697 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:13.377931 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:13.378354 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:13.378466 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:13.382379 (+ 3913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35405: BlockingRecv error: recv got EOF from 127.0.97.1:35405 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 03:07:13.383172 9705 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:13.374527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48145 (local address 127.0.97.62:42423)
0926 03:07:13.375012 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:13.375018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:13.375031 (+ 13us) server_negotiation.cc:407] Connection header received
0926 03:07:13.375071 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:13.375076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:13.375149 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:13.375235 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:13.375956 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:13.376925 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:13.378115 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:13.378762 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:13.378865 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:13.382633 (+ 3768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:48145: BlockingRecv error: recv got EOF from 127.0.97.3:48145 (error 108)
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":264,"threads_started":1}
Sep 26 03:07:13 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:13 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:13.909075 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:13.900243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.2:40337)
0926 03:07:13.900710 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:13.900737 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:13.900844 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:13.901196 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:13.901199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:13.901571 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:13.901745 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:13.901753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:13.903044 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:13.903048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:13.903948 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:13.903961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:13.904686 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:13.904691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:13.904827 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:13.905481 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:13.905499 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:13.908736 (+ 3237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":243,"threads_started":1}
W20260926 03:07:13.909226 9707 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:13.900332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40337 (local address 127.0.97.62:42423)
0926 03:07:13.900940 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:13.900945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:13.900960 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:07:13.901014 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:13.901017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:13.901073 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:13.901157 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:13.901978 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:13.902893 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:13.904095 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:13.904506 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:13.904616 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:13.908985 (+ 4369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40337: BlockingRecv error: recv got EOF from 127.0.97.2:40337 (error 108)
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":434,"threads_started":1}
Sep 26 03:07:14 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:14 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:14 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:14 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:14.397255 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:14.383678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.1:53229)
0926 03:07:14.387533 (+ 3855us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:14.387558 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:14.387781 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:14.388332 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:14.388335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:14.388853 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:14.389053 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:14.389064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:14.390704 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:14.390710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:14.392213 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:14.392227 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:14.393154 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:14.393160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:14.393339 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:14.393983 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:14.394007 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:14.397035 (+ 3028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3669,"thread_start_us":3537,"threads_started":1}
W20260926 03:07:14.397590 9707 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:14.383738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53229 (local address 127.0.97.62:42423)
0926 03:07:14.383911 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:14.383917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:14.387661 (+ 3744us) server_negotiation.cc:407] Connection header received
0926 03:07:14.387956 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:14.387962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:14.388039 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:14.388110 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:14.389233 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:14.390573 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:14.392352 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:14.392977 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:14.393164 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:14.397290 (+ 4126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53229: BlockingRecv error: recv got EOF from 127.0.97.1:53229 (error 108)
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":48}
I20260926 03:07:14.397742 9708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:14.383661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.3:53987)
0926 03:07:14.388023 (+ 4362us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:14.388040 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:14.388155 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:14.388483 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:14.388488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:14.388953 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:14.389189 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:14.389203 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:14.390704 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:14.390710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:14.392157 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:14.392173 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:14.393206 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:14.393211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:14.393409 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:14.394067 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:14.394083 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:14.397454 (+ 3371us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":4246,"thread_start_us":4126,"threads_started":1}
W20260926 03:07:14.398036 9710 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:14.383898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53987 (local address 127.0.97.62:42423)
0926 03:07:14.387882 (+ 3984us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:14.387887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:14.388119 (+ 232us) server_negotiation.cc:407] Connection header received
0926 03:07:14.388248 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:14.388252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:14.388312 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:14.388420 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:14.389384 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:14.390559 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:14.392304 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:14.392944 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:14.393073 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:14.397788 (+ 4715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53987: BlockingRecv error: recv got EOF from 127.0.97.3:53987 (error 108)
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":3857,"thread_start_us":3699,"threads_started":1}
Sep 26 03:07:14 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:14 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:14.919056 9711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:14.910098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.2:55565)
0926 03:07:14.910666 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:14.910684 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:14.910805 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:14.911200 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:14.911203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:14.911566 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:14.911732 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:14.911741 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:14.912950 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:14.912955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:14.913944 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:14.913957 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:14.914744 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:14.914751 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:14.914893 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:14.915657 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:14.915676 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:14.918824 (+ 3148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":297,"threads_started":1}
W20260926 03:07:14.919276 9712 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:14.910207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55565 (local address 127.0.97.62:42423)
0926 03:07:14.910668 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:14.910673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:14.910758 (+ 85us) server_negotiation.cc:407] Connection header received
0926 03:07:14.910956 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:14.910960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:14.911024 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:14.911125 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:14.911900 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:14.912745 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:14.914119 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:14.914581 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:14.914689 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:14.919032 (+ 4343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55565: BlockingRecv error: recv got EOF from 127.0.97.2:55565 (error 108)
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":281,"threads_started":1}
Sep 26 03:07:15 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:15 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:15 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:15 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:15.411656 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:15.398689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.3:52137)
0926 03:07:15.399401 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:15.399417 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:15.399539 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:15.399874 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:15.399879 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:15.400292 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:15.400482 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:15.400494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:15.401802 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:15.401807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:15.402897 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:15.402913 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:15.403827 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:15.403831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:15.403998 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:15.408442 (+ 4444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:15.408467 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:15.411425 (+ 2958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":476,"threads_started":1}
W20260926 03:07:15.412007 9715 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:15.398720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52137 (local address 127.0.97.62:42423)
0926 03:07:15.399250 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:15.399255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:15.399498 (+ 243us) server_negotiation.cc:407] Connection header received
0926 03:07:15.399661 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:15.399666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:15.399730 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:15.399814 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:15.400651 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:15.401635 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:15.403086 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:15.403684 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:15.403803 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:15.411712 (+ 7909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52137: BlockingRecv error: recv got EOF from 127.0.97.3:52137 (error 108)
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":422,"thread_start_us":276,"threads_started":1}
I20260926 03:07:15.412093 9713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:15.398286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.1:54431)
0926 03:07:15.398853 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:15.398876 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:15.399095 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:15.399568 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:15.399571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:15.400026 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:15.400177 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:15.400186 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:15.401316 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:15.401322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:15.402414 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:15.402435 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:15.403383 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:15.403388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:15.403584 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:15.408393 (+ 4809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:15.408418 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:15.411685 (+ 3267us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":374,"threads_started":1}
W20260926 03:07:15.412254 9712 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:15.398473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54431 (local address 127.0.97.62:42423)
0926 03:07:15.398664 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:15.398670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:15.398972 (+ 302us) server_negotiation.cc:407] Connection header received
0926 03:07:15.399250 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:15.399255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:15.399329 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:15.399397 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:15.400308 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:15.401150 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:15.402593 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:15.403210 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:15.403297 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:15.411919 (+ 8622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54431: BlockingRecv error: recv got EOF from 127.0.97.1:54431 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":36}
Sep 26 03:07:15 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:15 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:15.928156 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:15.919919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.2:48479)
0926 03:07:15.920470 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:15.920491 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:15.920626 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:15.920992 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:15.920995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:15.921421 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:15.921585 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:15.921594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:15.922687 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:15.922691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:15.923614 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:15.923627 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:15.924331 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:15.924335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:15.924445 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:15.925004 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:15.925023 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:15.927914 (+ 2891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":240,"threads_started":1}
W20260926 03:07:15.928414 9717 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:15.920160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48479 (local address 127.0.97.62:42423)
0926 03:07:15.920630 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:15.920636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:15.920654 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:07:15.920765 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:15.920769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:15.920826 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:15.920913 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:15.921736 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:15.922558 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:15.923775 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:15.924185 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:15.924278 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:15.928171 (+ 3893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48479: BlockingRecv error: recv got EOF from 127.0.97.2:48479 (error 108)
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":228,"threads_started":1}
Sep 26 03:07:16 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:16 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:16 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:16 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:16.422159 9719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:16.413075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.1:36487)
0926 03:07:16.413684 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:16.413706 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:16.413913 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:16.414417 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:16.414420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:16.414809 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:16.414981 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:16.414990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:16.416081 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:16.416085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:16.417006 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:16.417017 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:16.417745 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:16.417750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:16.417934 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:16.418754 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:16.418778 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:16.421922 (+ 3144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":324,"threads_started":1}
W20260926 03:07:16.422406 9717 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:16.413111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36487 (local address 127.0.97.62:42423)
0926 03:07:16.413283 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:16.413288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:16.413798 (+ 510us) server_negotiation.cc:407] Connection header received
0926 03:07:16.414071 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:16.414077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:16.414173 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:16.414231 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:16.415140 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:16.415931 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:16.417138 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:16.417556 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:16.417706 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:16.422176 (+ 4470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36487: BlockingRecv error: recv got EOF from 127.0.97.1:36487 (error 108)
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":54}
I20260926 03:07:16.422762 9718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:16.412867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.3:34135)
0926 03:07:16.413950 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:16.413970 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:16.414092 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:16.414611 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:16.414617 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:16.415195 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:16.415359 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:16.415369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:16.416579 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:16.416583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:16.417505 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:16.417520 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:16.418595 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:16.418601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:16.418753 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:16.419501 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:16.419525 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:16.422423 (+ 2898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":975,"thread_start_us":846,"threads_started":1}
W20260926 03:07:16.422946 9720 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:16.413197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34135 (local address 127.0.97.62:42423)
0926 03:07:16.414317 (+ 1120us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:16.414321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:16.414335 (+ 14us) server_negotiation.cc:407] Connection header received
0926 03:07:16.414372 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:16.414375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:16.414422 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:16.414535 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:16.415544 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:16.416433 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:16.417652 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:16.418413 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:16.418593 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:16.422697 (+ 4104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34135: BlockingRecv error: recv got EOF from 127.0.97.3:34135 (error 108)
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1009,"thread_start_us":774,"threads_started":1}
Sep 26 03:07:16 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:16 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:16.938333 9721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:16.929201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.2:37113)
0926 03:07:16.930065 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:16.930083 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:16.930196 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:16.930651 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:16.930657 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:16.931017 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:16.931201 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:16.931210 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:16.932264 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:16.932268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:16.933208 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:16.933222 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:16.934031 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:16.934038 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:16.934183 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:16.934850 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:16.934868 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:16.938081 (+ 3213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":307,"threads_started":1}
W20260926 03:07:16.938634 9722 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:16.929342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37113 (local address 127.0.97.62:42423)
0926 03:07:16.929985 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:16.929992 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:16.930155 (+ 163us) server_negotiation.cc:407] Connection header received
0926 03:07:16.930358 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:16.930363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:16.930438 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:16.930579 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:16.931356 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:16.932104 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:16.933368 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:16.933811 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:16.933920 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:16.938345 (+ 4425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37113: BlockingRecv error: recv got EOF from 127.0.97.2:37113 (error 108)
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":280,"threads_started":1}
Sep 26 03:07:17 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:17 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:17 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:17 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:17.433511 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:17.423423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.1:59223)
0926 03:07:17.424383 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:17.424405 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:17.424544 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:17.425246 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:17.425251 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:17.426036 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:17.426331 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:17.426342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:17.427448 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:17.427454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:17.428390 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:17.428413 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:17.429059 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:17.429065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:17.429237 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:17.429853 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:17.429870 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:17.433158 (+ 3288us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":829,"thread_start_us":723,"threads_started":1}
W20260926 03:07:17.433674 9722 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:17.423592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59223 (local address 127.0.97.62:42423)
0926 03:07:17.424511 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:17.424517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:17.424538 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:07:17.424696 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:17.424701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:17.424780 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:17.424854 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:17.426474 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:17.427313 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:17.428531 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:17.428915 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:17.429026 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:17.433406 (+ 4380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59223: BlockingRecv error: recv got EOF from 127.0.97.1:59223 (error 108)
Metrics: {"server-negotiator.queue_time_us":763}
I20260926 03:07:17.433734 9724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:17.423656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.3:52365)
0926 03:07:17.425555 (+ 1899us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:17.425574 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:17.425684 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:17.426179 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:17.426184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:17.426595 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:17.426763 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:17.426773 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:17.427811 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:17.427815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:17.428910 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:17.428926 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:17.429742 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:17.429749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:17.429908 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:17.430532 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:17.430556 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:17.433444 (+ 2888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1767,"thread_start_us":1425,"threads_started":1}
W20260926 03:07:17.433933 9725 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:17.423719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52365 (local address 127.0.97.62:42423)
0926 03:07:17.425642 (+ 1923us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:17.425647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:17.425667 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:07:17.425959 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:17.425963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:17.426014 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:17.426224 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:17.426906 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:17.427668 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:17.429081 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:17.429590 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:17.429694 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:17.433704 (+ 4010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52365: BlockingRecv error: recv got EOF from 127.0.97.3:52365 (error 108)
Metrics: {"server-negotiator.queue_time_us":1831,"thread_start_us":1696,"threads_started":1}
Sep 26 03:07:17 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:07:17 dist-test-slave-hmbw krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:07:17.948874 9726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:17.939591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42423 (local address 127.0.97.2:51423)
0926 03:07:17.940065 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:17.940082 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:17.940197 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:17.940713 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:17.940716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:17.941191 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:17.941403 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:17.941413 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:17.942587 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:17.942591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:17.943572 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:17.943585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:17.944424 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:17.944430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:17.944564 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:17.945248 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:17.945266 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:17.948581 (+ 3315us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42423: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":246,"threads_started":1}
W20260926 03:07:17.949527 9727 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:17.939649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51423 (local address 127.0.97.62:42423)
0926 03:07:17.940055 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:17.940061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:17.940240 (+ 179us) server_negotiation.cc:407] Connection header received
0926 03:07:17.940386 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:17.940390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:17.940514 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:17.940661 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:17.941597 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:17.942387 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:17.943740 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:17.944236 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:17.944347 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:17.949083 (+ 4736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51423: BlockingRecv error: recv got EOF from 127.0.97.2:51423 (error 108)
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":228,"threads_started":1}
I20260926 03:07:18.260010 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 9240
I20260926 03:07:18.271804 9729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:18.261153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33014 (local address 127.0.97.1:44599)
0926 03:07:18.261653 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:18.261660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:18.261685 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:07:18.261749 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:18.261756 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:18.261970 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:18.262129 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:18.263614 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:18.264630 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:18.265863 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:18.266438 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:18.266614 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:18.268957 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:18.268966 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:18.268981 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:07:18.269013 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:18.271312 (+ 2299us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 03:07:18.271580 (+ 268us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33014: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":221,"threads_started":1}
W20260926 03:07:18.271907 9729 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33014: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 03:07:18.272111 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 9240: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:44599: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 03:07:18.272280 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 9240
I20260926 03:07:18.286377 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 9391
I20260926 03:07:18.298878 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:18.287414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38282 (local address 127.0.97.2:41937)
0926 03:07:18.287990 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:18.287997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:18.288026 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:07:18.288083 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:18.288089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:18.288355 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:18.288517 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:18.289836 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:18.290961 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:18.292164 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:18.292715 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:18.292834 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:18.295500 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:18.295509 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:18.295524 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:07:18.295579 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:18.298144 (+ 2565us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 03:07:18.298475 (+ 331us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38282: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":281,"threads_started":1}
W20260926 03:07:18.298985 9730 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38282: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 03:07:18.298995 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 9391: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:41937: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 03:07:18.299139 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 9391
I20260926 03:07:18.312847 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 9523
I20260926 03:07:18.325196 9731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:18.313781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47864 (local address 127.0.97.3:41513)
0926 03:07:18.314435 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:18.314442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:18.314465 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:07:18.314637 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:18.314645 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:18.314874 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:18.315031 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:18.316373 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:18.317478 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:18.318771 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:18.319430 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:18.319550 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:18.322076 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:18.322087 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:18.322102 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:07:18.322150 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:18.324656 (+ 2506us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 03:07:18.324940 (+ 284us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47864: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":387,"threads_started":1}
W20260926 03:07:18.325294 9731 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47864: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 03:07:18.325515 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 9523: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:41513: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 03:07:18.325677 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 9523
I20260926 03:07:18.339206 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 9171
I20260926 03:07:18.340513 9233 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:07:18.454351 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 9171
2026-09-26T03:07:18Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21247 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T03:07:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:07:18Z Disabled control of system clock
I20260926 03:07:18.526994 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39689
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 03:07:18.849772 9736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:07:18.850070 9736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:07:18.850167 9736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:07:18.857555 9736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:07:18.857673 9736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:07:18.857710 9736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:07:18.857784 9736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:07:18.857844 9736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:07:18.867408 9736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39689
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:07:18.868963 9736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:07:18.870854 9736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:07:18.883093 9742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:18.883112 9744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:18.883147 9741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:07:18.883230 9736 server_base.cc:1034] running on GCE node
I20260926 03:07:18.884377 9736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:07:18.885797 9736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:07:18.887064 9736 hybrid_clock.cc:648] HybridClock initialized: now 1790392038886957 us; error 50 us; skew 500 ppm
I20260926 03:07:18.887416 9736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:07:18.889791 9736 webserver.cc:460] Webserver started at http://127.0.97.62:33643/ using document root <none> and password file <none>
I20260926 03:07:18.890563 9736 fs_manager.cc:359] Metadata directory not provided
I20260926 03:07:18.890690 9736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:07:18.891028 9736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:07:18.893945 9736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "4240c7812249416aa0a716ac042c20eb"
format_stamp: "Formatted at 2026-09-26 03:07:18 on dist-test-slave-hmbw"
server_key: "e275e71cc0f0f240dd88e8bf06f9b7a5"
server_key_iv: "ddc67672e7173ddad62fd8c2c853f881"
server_key_version: "encryptionkey@0"
I20260926 03:07:18.894781 9736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "4240c7812249416aa0a716ac042c20eb"
format_stamp: "Formatted at 2026-09-26 03:07:18 on dist-test-slave-hmbw"
server_key: "e275e71cc0f0f240dd88e8bf06f9b7a5"
server_key_iv: "ddc67672e7173ddad62fd8c2c853f881"
server_key_version: "encryptionkey@0"
I20260926 03:07:18.899999 9736 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 03:07:18.903513 9750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:18.904970 9736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:07:18.905141 9736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "4240c7812249416aa0a716ac042c20eb"
format_stamp: "Formatted at 2026-09-26 03:07:18 on dist-test-slave-hmbw"
server_key: "e275e71cc0f0f240dd88e8bf06f9b7a5"
server_key_iv: "ddc67672e7173ddad62fd8c2c853f881"
server_key_version: "encryptionkey@0"
I20260926 03:07:18.905350 9736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:07:18.934847 9736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:07:18.935760 9736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:07:18.936054 9736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:07:18.953851 9736 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39689
I20260926 03:07:18.953871 9801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39689 every 8 connection(s)
I20260926 03:07:18.955252 9736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:07:18.959044 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 9736
I20260926 03:07:18.959121 9802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:07:18.959218 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:07:18.959929 388 external_mini_cluster.cc:1428] Setting key c85fcd36eadad86af7a2c2952cd39d8f
I20260926 03:07:18.973654 9802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4240c7812249416aa0a716ac042c20eb: Bootstrap starting.
I20260926 03:07:18.976580 9805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:18.962057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43788 (local address 127.0.97.62:39689)
0926 03:07:18.963192 (+ 1135us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:18.963207 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:18.963257 (+ 50us) server_negotiation.cc:407] Connection header received
0926 03:07:18.965556 (+ 2299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:18.965589 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:18.966327 (+ 738us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:18.966843 (+ 516us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:18.967397 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:18.969255 (+ 1858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:18.971057 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:18.972229 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:18.972452 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:18.973660 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:18.973672 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:18.973692 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:07:18.974407 (+ 715us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:18.974541 (+ 134us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:07:18.974572 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:07:18.974686 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:18.974931 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 03:07:18.975290 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":347,"threads_started":1}
I20260926 03:07:18.977588 9802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4240c7812249416aa0a716ac042c20eb: Neither blocks nor log segments found. Creating new log.
I20260926 03:07:18.978883 9802 log.cc:824] T 00000000000000000000000000000000 P 4240c7812249416aa0a716ac042c20eb: Log is configured to *not* fsync() on all Append() calls
I20260926 03:07:18.981935 9802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4240c7812249416aa0a716ac042c20eb: No bootstrap required, opened a new log
I20260926 03:07:18.989594 9802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4240c7812249416aa0a716ac042c20eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4240c7812249416aa0a716ac042c20eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39689 } }
I20260926 03:07:18.990000 9802 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4240c7812249416aa0a716ac042c20eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:07:18.990078 9802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4240c7812249416aa0a716ac042c20eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4240c7812249416aa0a716ac042c20eb, State: Initialized, Role: FOLLOWER
I20260926 03:07:18.990638 9802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4240c7812249416aa0a716ac042c20eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4240c7812249416aa0a716ac042c20eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39689 } }
I20260926 03:07:18.990875 9802 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4240c7812249416aa0a716ac042c20eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:07:18.991010 9802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4240c7812249416aa0a716ac042c20eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:07:18.991189 9802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4240c7812249416aa0a716ac042c20eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:07:18.993582 9802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4240c7812249416aa0a716ac042c20eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4240c7812249416aa0a716ac042c20eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39689 } }
I20260926 03:07:18.994125 9802 leader_election.cc:302] T 00000000000000000000000000000000 P 4240c7812249416aa0a716ac042c20eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4240c7812249416aa0a716ac042c20eb; no voters:
I20260926 03:07:18.994752 9802 leader_election.cc:288] T 00000000000000000000000000000000 P 4240c7812249416aa0a716ac042c20eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:07:18.994881 9807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4240c7812249416aa0a716ac042c20eb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:07:18.995599 9807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4240c7812249416aa0a716ac042c20eb [term 1 LEADER]: Becoming Leader. State: Replica: 4240c7812249416aa0a716ac042c20eb, State: Running, Role: LEADER
I20260926 03:07:18.996192 9807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4240c7812249416aa0a716ac042c20eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4240c7812249416aa0a716ac042c20eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39689 } }
I20260926 03:07:18.996848 9802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4240c7812249416aa0a716ac042c20eb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:07:18.999109 9808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4240c7812249416aa0a716ac042c20eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4240c7812249416aa0a716ac042c20eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4240c7812249416aa0a716ac042c20eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39689 } } }
I20260926 03:07:18.999692 9808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4240c7812249416aa0a716ac042c20eb [sys.catalog]: This master's current role is: LEADER
I20260926 03:07:18.999262 9809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4240c7812249416aa0a716ac042c20eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4240c7812249416aa0a716ac042c20eb. Latest consensus state: current_term: 1 leader_uuid: "4240c7812249416aa0a716ac042c20eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4240c7812249416aa0a716ac042c20eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39689 } } }
I20260926 03:07:19.000198 9809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4240c7812249416aa0a716ac042c20eb [sys.catalog]: This master's current role is: LEADER
I20260926 03:07:19.001435 9816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:07:19.006474 9816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:07:19.017140 9816 catalog_manager.cc:1347] Generated new cluster ID: 49f2a1371a7c4de3920975a04a4d9a44
I20260926 03:07:19.017256 9816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:07:19.044530 9816 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:07:19.045898 9816 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:07:19.058902 9816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4240c7812249416aa0a716ac042c20eb: Generated new TSK 0
I20260926 03:07:19.059839 9816 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:07:19.067896 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39689
--builtin_ntp_servers=127.0.97.20:33457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 03:07:19.420235 9826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:07:19.420523 9826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:07:19.420629 9826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:07:19.427974 9826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:07:19.428097 9826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:07:19.428234 9826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:07:19.437965 9826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:07:19.439453 9826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:07:19.441303 9826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:07:19.454315 9831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:19.454335 9832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:19.454325 9834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:07:19.455092 9826 server_base.cc:1034] running on GCE node
I20260926 03:07:19.455754 9826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:07:19.456720 9826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:07:19.457963 9826 hybrid_clock.cc:648] HybridClock initialized: now 1790392039457875 us; error 82 us; skew 500 ppm
I20260926 03:07:19.458365 9826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:07:19.460789 9826 webserver.cc:460] Webserver started at http://127.0.97.1:43639/ using document root <none> and password file <none>
I20260926 03:07:19.461527 9826 fs_manager.cc:359] Metadata directory not provided
I20260926 03:07:19.461632 9826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:07:19.461949 9826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:07:19.464995 9826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/data/instance:
uuid: "e6110932a0cf4bf3af8ed430dbf1fa02"
format_stamp: "Formatted at 2026-09-26 03:07:19 on dist-test-slave-hmbw"
server_key: "d8ac16d898435e57deb668edad9da297"
server_key_iv: "65e37108880b7afafbb43818c29cd2ee"
server_key_version: "encryptionkey@0"
I20260926 03:07:19.465849 9826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e6110932a0cf4bf3af8ed430dbf1fa02"
format_stamp: "Formatted at 2026-09-26 03:07:19 on dist-test-slave-hmbw"
server_key: "d8ac16d898435e57deb668edad9da297"
server_key_iv: "65e37108880b7afafbb43818c29cd2ee"
server_key_version: "encryptionkey@0"
I20260926 03:07:19.471328 9826 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 03:07:19.475142 9840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:19.476570 9826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:07:19.476725 9826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "e6110932a0cf4bf3af8ed430dbf1fa02"
format_stamp: "Formatted at 2026-09-26 03:07:19 on dist-test-slave-hmbw"
server_key: "d8ac16d898435e57deb668edad9da297"
server_key_iv: "65e37108880b7afafbb43818c29cd2ee"
server_key_version: "encryptionkey@0"
I20260926 03:07:19.476941 9826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:07:19.489626 9826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:07:19.490605 9826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:07:19.490860 9826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:07:19.491874 9826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:07:19.493654 9826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:07:19.493758 9826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:19.493870 9826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:07:19.493937 9826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:19.522411 9826 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41821
I20260926 03:07:19.522441 9952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41821 every 8 connection(s)
I20260926 03:07:19.523734 9826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:07:19.532851 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 9826
I20260926 03:07:19.533025 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:07:19.533613 388 external_mini_cluster.cc:1428] Setting key f2863cf2b269747df49c42c787b788bd
I20260926 03:07:19.536095 9955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:19.526325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34809 (local address 127.0.97.62:39689)
0926 03:07:19.526780 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:19.526785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:19.527870 (+ 1085us) server_negotiation.cc:407] Connection header received
0926 03:07:19.529642 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:19.529648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:19.529729 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:19.529819 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:19.530988 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:19.531691 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:19.533276 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:19.533675 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:19.533801 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:19.535332 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:19.535338 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:19.535343 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:07:19.535365 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:19.535414 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:07:19.535429 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:07:19.535517 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:19.535762 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 03:07:19.535839 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":252,"threads_started":1}
I20260926 03:07:19.536597 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39689
--builtin_ntp_servers=127.0.97.20:33457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 03:07:19.537518 9956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:19.526811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39689 (local address 127.0.97.1:34809)
0926 03:07:19.527700 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:19.527742 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:19.529332 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:19.530071 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:19.530080 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:19.530178 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:19.530803 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:19.530823 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:19.531845 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:19.531849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:19.533137 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:19.533153 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:19.533918 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:19.533926 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:19.534285 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:19.535020 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:07:19.535022 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:19.535067 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:07:19.535146 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:07:19.535530 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:07:19.535538 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:07:19.535545 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 03:07:19.535738 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 03:07:19.536173 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":267,"threads_started":1}
I20260926 03:07:19.539312 9953 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39689
I20260926 03:07:19.539618 9953 heartbeater.cc:464] Registering TS with master...
I20260926 03:07:19.540369 9953 heartbeater.cc:511] Master 127.0.97.62:39689 requested a full tablet report, sending...
I20260926 03:07:19.542728 9767 ts_manager.cc:194] Registered new tserver with Master: e6110932a0cf4bf3af8ed430dbf1fa02 (127.0.97.1:41821)
I20260926 03:07:19.544543 9767 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:34809
W20260926 03:07:19.859225 9957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:07:19.859513 9957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:07:19.859607 9957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:07:19.867254 9957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:07:19.867379 9957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:07:19.867486 9957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:07:19.877086 9957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:07:19.878360 9957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:07:19.880283 9957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:07:19.892939 9962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:19.893239 9965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:19.893271 9963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:07:19.894204 9957 server_base.cc:1034] running on GCE node
I20260926 03:07:19.894915 9957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:07:19.895936 9957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:07:19.897181 9957 hybrid_clock.cc:648] HybridClock initialized: now 1790392039897120 us; error 45 us; skew 500 ppm
I20260926 03:07:19.897500 9957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:07:19.899904 9957 webserver.cc:460] Webserver started at http://127.0.97.2:38871/ using document root <none> and password file <none>
I20260926 03:07:19.900663 9957 fs_manager.cc:359] Metadata directory not provided
I20260926 03:07:19.900774 9957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:07:19.901086 9957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:07:19.904042 9957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/data/instance:
uuid: "bda8fc72221448a3adaed0e991e0f012"
format_stamp: "Formatted at 2026-09-26 03:07:19 on dist-test-slave-hmbw"
server_key: "1279ace50b7776a0d2434cb57377f7e5"
server_key_iv: "cf8fa851318fc0fe2c3115ebba9349ef"
server_key_version: "encryptionkey@0"
I20260926 03:07:19.904829 9957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/wal/instance:
uuid: "bda8fc72221448a3adaed0e991e0f012"
format_stamp: "Formatted at 2026-09-26 03:07:19 on dist-test-slave-hmbw"
server_key: "1279ace50b7776a0d2434cb57377f7e5"
server_key_iv: "cf8fa851318fc0fe2c3115ebba9349ef"
server_key_version: "encryptionkey@0"
I20260926 03:07:19.910135 9957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 03:07:19.913705 9971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:19.915252 9957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 03:07:19.915407 9957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "bda8fc72221448a3adaed0e991e0f012"
format_stamp: "Formatted at 2026-09-26 03:07:19 on dist-test-slave-hmbw"
server_key: "1279ace50b7776a0d2434cb57377f7e5"
server_key_iv: "cf8fa851318fc0fe2c3115ebba9349ef"
server_key_version: "encryptionkey@0"
I20260926 03:07:19.915608 9957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:07:19.944945 9957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:07:19.945853 9957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:07:19.946110 9957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:07:19.947080 9957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:07:19.948644 9957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:07:19.948737 9957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:19.948840 9957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:07:19.948910 9957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:19.981119 9957 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40121
I20260926 03:07:19.981178 10083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40121 every 8 connection(s)
I20260926 03:07:19.982460 9957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:07:19.989531 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 9957
I20260926 03:07:19.989693 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:07:19.990125 388 external_mini_cluster.cc:1428] Setting key 385386cf215d5c8af869669f595dddcf
I20260926 03:07:19.993149 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39689
--builtin_ntp_servers=127.0.97.20:33457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 03:07:19.996157 9955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:19.985066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48273 (local address 127.0.97.62:39689)
0926 03:07:19.985264 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:19.985270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:19.986971 (+ 1701us) server_negotiation.cc:407] Connection header received
0926 03:07:19.989052 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:19.989057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:19.989161 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:19.989263 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:19.990619 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:19.991474 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:19.993227 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:19.993674 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:19.993786 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:19.995363 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:19.995370 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:19.995374 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:07:19.995404 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:19.995439 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:07:19.995453 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:07:19.995528 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:19.995792 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 03:07:19.995887 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:07:19.997610 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:19.985663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39689 (local address 127.0.97.2:48273)
0926 03:07:19.986750 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:19.986792 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:19.988739 (+ 1947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:19.989536 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:19.989546 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:19.989658 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:19.990354 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:19.990376 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:19.991636 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:19.991640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:19.993068 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:19.993090 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:19.993906 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:19.993914 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:19.994151 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:19.994968 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:07:19.994972 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:19.995029 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:07:19.995123 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:07:19.995567 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:07:19.995579 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:07:19.995588 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 03:07:19.995796 (+ 208us) client_negotiation.cc:240] Negotiation successful
0926 03:07:19.996230 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":334,"threads_started":1}
I20260926 03:07:19.999190 10084 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39689
I20260926 03:07:19.999532 10084 heartbeater.cc:464] Registering TS with master...
I20260926 03:07:20.000407 10084 heartbeater.cc:511] Master 127.0.97.62:39689 requested a full tablet report, sending...
I20260926 03:07:20.001974 9767 ts_manager.cc:194] Registered new tserver with Master: bda8fc72221448a3adaed0e991e0f012 (127.0.97.2:40121)
I20260926 03:07:20.002859 9767 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:48273
W20260926 03:07:20.345865 10087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:07:20.346194 10087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:07:20.346298 10087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:07:20.353720 10087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:07:20.353848 10087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:07:20.353976 10087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:07:20.363359 10087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:07:20.364743 10087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:07:20.367317 10087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:07:20.382047 10093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:20.382038 10092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:20.382228 10095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:07:20.382911 10087 server_base.cc:1034] running on GCE node
I20260926 03:07:20.383553 10087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:07:20.384523 10087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:07:20.385746 10087 hybrid_clock.cc:648] HybridClock initialized: now 1790392040385694 us; error 45 us; skew 500 ppm
I20260926 03:07:20.386122 10087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:07:20.388583 10087 webserver.cc:460] Webserver started at http://127.0.97.3:37615/ using document root <none> and password file <none>
I20260926 03:07:20.389282 10087 fs_manager.cc:359] Metadata directory not provided
I20260926 03:07:20.389380 10087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:07:20.389683 10087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:07:20.392550 10087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/data/instance:
uuid: "8e1f22596beb463985413be797aee2db"
format_stamp: "Formatted at 2026-09-26 03:07:20 on dist-test-slave-hmbw"
server_key: "e8347b19af2ff2887bc8004a1d8bb0eb"
server_key_iv: "02f231644bd38ad477f6d973b467b75a"
server_key_version: "encryptionkey@0"
I20260926 03:07:20.393334 10087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8e1f22596beb463985413be797aee2db"
format_stamp: "Formatted at 2026-09-26 03:07:20 on dist-test-slave-hmbw"
server_key: "e8347b19af2ff2887bc8004a1d8bb0eb"
server_key_iv: "02f231644bd38ad477f6d973b467b75a"
server_key_version: "encryptionkey@0"
I20260926 03:07:20.399149 10087 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 03:07:20.402987 10101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:20.404500 10087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:07:20.404681 10087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "8e1f22596beb463985413be797aee2db"
format_stamp: "Formatted at 2026-09-26 03:07:20 on dist-test-slave-hmbw"
server_key: "e8347b19af2ff2887bc8004a1d8bb0eb"
server_key_iv: "02f231644bd38ad477f6d973b467b75a"
server_key_version: "encryptionkey@0"
I20260926 03:07:20.404902 10087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:07:20.426633 10087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:07:20.427561 10087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:07:20.427843 10087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:07:20.428824 10087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:07:20.430368 10087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:07:20.430465 10087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:20.430626 10087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:07:20.430697 10087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:20.461786 10087 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41447
I20260926 03:07:20.461839 10213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41447 every 8 connection(s)
I20260926 03:07:20.463353 10087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:07:20.468988 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 10087
I20260926 03:07:20.469127 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:07:20.469550 388 external_mini_cluster.cc:1428] Setting key c21e51338505d8a251e22a6037a19ac1
I20260926 03:07:20.476850 9955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:20.465951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53231 (local address 127.0.97.62:39689)
0926 03:07:20.466191 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:20.466197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:20.467693 (+ 1496us) server_negotiation.cc:407] Connection header received
0926 03:07:20.469671 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:20.469677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:20.469767 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:20.469852 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:20.471163 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:20.472171 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:20.473916 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:20.474314 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:20.474417 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:20.475987 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:20.475993 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:20.476001 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:07:20.476022 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:20.476053 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:07:20.476065 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:07:20.476139 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:20.476494 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 03:07:20.476581 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260926 03:07:20.477886 10216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:20.466443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39689 (local address 127.0.97.3:53231)
0926 03:07:20.467498 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:20.467543 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:20.469365 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:20.470113 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:20.470122 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:20.470222 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:20.470999 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:20.471018 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:20.472348 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:20.472353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:20.473766 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:20.473780 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:20.474563 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:20.474570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:20.474800 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:20.475644 (+ 844us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:07:20.475647 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:20.475689 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:07:20.475774 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:07:20.476188 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:07:20.476199 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:07:20.476206 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 03:07:20.476379 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 03:07:20.476730 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":314,"threads_started":1}
I20260926 03:07:20.479370 10214 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39689
I20260926 03:07:20.479688 10214 heartbeater.cc:464] Registering TS with master...
I20260926 03:07:20.480600 10214 heartbeater.cc:511] Master 127.0.97.62:39689 requested a full tablet report, sending...
I20260926 03:07:20.482257 9767 ts_manager.cc:194] Registered new tserver with Master: 8e1f22596beb463985413be797aee2db (127.0.97.3:41447)
I20260926 03:07:20.483219 9767 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:53231
I20260926 03:07:20.485221 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 03:07:20.491482 9955 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:20.489863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43800 (local address 127.0.97.62:39689)
0926 03:07:20.490055 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:20.490062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:20.490447 (+ 385us) server_negotiation.cc:407] Connection header received
0926 03:07:20.490631 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:20.490637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:20.490714 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:20.490796 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:20.491225 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43800: BlockingRecv error: recv got EOF from 127.0.0.1:43800 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:07:20.492216 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 9826
I20260926 03:07:20.500626 10223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:20.493176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49148 (local address 127.0.97.1:41821)
0926 03:07:20.493750 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:20.493757 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:20.493783 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:07:20.493837 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:20.493843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:20.494058 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:20.494187 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:20.494630 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:20.495791 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:20.497247 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:20.497878 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:20.497996 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:20.499134 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:20.499142 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:20.499159 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:07:20.499814 (+ 655us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:20.499866 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:07:20.499913 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:07:20.500035 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:20.500183 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 03:07:20.500307 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":334,"threads_started":1}
I20260926 03:07:20.501940 9948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:07:20.583257 9953 heartbeater.cc:503] Master 127.0.97.62:39689 was elected leader, sending a full tablet report...
I20260926 03:07:20.584642 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 9826
W20260926 03:07:20.598600 9751 connection.cc:535] server connection from 127.0.97.1:34809 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 03:07:20.598773 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 9957
I20260926 03:07:20.606283 10225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:20.599592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35050 (local address 127.0.97.2:40121)
0926 03:07:20.600094 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:20.600101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:20.600126 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:07:20.600183 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:20.600189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:20.600433 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:20.600575 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:20.600954 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:20.601964 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:20.603220 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:20.603768 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:20.603883 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:20.604875 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:20.604883 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:20.604897 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:07:20.605489 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:20.605537 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:07:20.605557 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:07:20.605675 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:20.605801 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 03:07:20.605941 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":249,"threads_started":1}
I20260926 03:07:20.607703 10079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:07:20.689477 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 9957
I20260926 03:07:20.703650 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 10087
I20260926 03:07:20.712209 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:20.704727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38872 (local address 127.0.97.3:41447)
0926 03:07:20.705368 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:20.705375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:20.705411 (+ 36us) server_negotiation.cc:407] Connection header received
0926 03:07:20.705476 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:20.705482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:20.705690 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:20.705826 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:20.706331 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:20.707494 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:20.708926 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:20.709570 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:20.709696 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:20.710813 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:20.710821 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:20.710835 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:07:20.711426 (+ 591us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:20.711480 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:07:20.711500 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:07:20.711644 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:20.711773 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 03:07:20.711895 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":345,"threads_started":1}
I20260926 03:07:20.713526 10209 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:07:20.794878 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 10087
I20260926 03:07:20.808003 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 9736
I20260926 03:07:20.809303 9797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:07:20.908350 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 9736
2026-09-26T03:07:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2448 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T03:07:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:07:20Z Disabled control of system clock
I20260926 03:07:20.976809 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32883
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 03:07:21.314199 10233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:07:21.314584 10233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:07:21.314710 10233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:07:21.322372 10233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:07:21.322536 10233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:07:21.322593 10233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:07:21.322676 10233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:07:21.322733 10233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:07:21.332340 10233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32883
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:07:21.333665 10233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:07:21.335517 10233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:07:21.347266 10241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:21.347393 10238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:21.347383 10239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:07:21.348299 10233 server_base.cc:1034] running on GCE node
I20260926 03:07:21.349123 10233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:07:21.350644 10233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:07:21.351914 10233 hybrid_clock.cc:648] HybridClock initialized: now 1790392041351851 us; error 59 us; skew 500 ppm
I20260926 03:07:21.352391 10233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:07:21.354825 10233 webserver.cc:460] Webserver started at http://127.0.97.62:44395/ using document root <none> and password file <none>
I20260926 03:07:21.355533 10233 fs_manager.cc:359] Metadata directory not provided
I20260926 03:07:21.355621 10233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:07:21.355904 10233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:07:21.358973 10233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "71ff04d583f64efbb959eca86fd34969"
format_stamp: "Formatted at 2026-09-26 03:07:21 on dist-test-slave-hmbw"
server_key: "645441560ab4c8a08734dd7073e8d2e9"
server_key_iv: "89c532a8188c9456d8e93a4c10c0d3df"
server_key_version: "encryptionkey@0"
I20260926 03:07:21.359756 10233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "71ff04d583f64efbb959eca86fd34969"
format_stamp: "Formatted at 2026-09-26 03:07:21 on dist-test-slave-hmbw"
server_key: "645441560ab4c8a08734dd7073e8d2e9"
server_key_iv: "89c532a8188c9456d8e93a4c10c0d3df"
server_key_version: "encryptionkey@0"
I20260926 03:07:21.365134 10233 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 03:07:21.368901 10247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:21.370430 10233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:07:21.370642 10233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "71ff04d583f64efbb959eca86fd34969"
format_stamp: "Formatted at 2026-09-26 03:07:21 on dist-test-slave-hmbw"
server_key: "645441560ab4c8a08734dd7073e8d2e9"
server_key_iv: "89c532a8188c9456d8e93a4c10c0d3df"
server_key_version: "encryptionkey@0"
I20260926 03:07:21.370877 10233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:07:21.376444 10233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:07:21.377260 10233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:07:21.377544 10233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:07:21.394975 10233 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32883
I20260926 03:07:21.395068 10298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32883 every 8 connection(s)
I20260926 03:07:21.396317 10233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:07:21.399250 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 10233
I20260926 03:07:21.399425 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:07:21.400054 388 external_mini_cluster.cc:1428] Setting key 4e7e6b7c209ee28aad1ef75a59c2f8c3
I20260926 03:07:21.400154 10299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:07:21.408900 10302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:21.402011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57522 (local address 127.0.97.62:32883)
0926 03:07:21.403112 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:21.403126 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:21.403176 (+ 50us) server_negotiation.cc:407] Connection header received
0926 03:07:21.404845 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:21.404873 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:21.405441 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:21.405883 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:21.406912 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:21.406923 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:21.406943 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:07:21.406960 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:21.407046 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:07:21.407068 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:07:21.407293 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:21.407428 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 03:07:21.407708 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":343,"threads_started":1}
I20260926 03:07:21.414297 10299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71ff04d583f64efbb959eca86fd34969: Bootstrap starting.
I20260926 03:07:21.417402 10299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71ff04d583f64efbb959eca86fd34969: Neither blocks nor log segments found. Creating new log.
I20260926 03:07:21.418362 10299 log.cc:824] T 00000000000000000000000000000000 P 71ff04d583f64efbb959eca86fd34969: Log is configured to *not* fsync() on all Append() calls
I20260926 03:07:21.421213 10299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71ff04d583f64efbb959eca86fd34969: No bootstrap required, opened a new log
I20260926 03:07:21.428730 10299 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71ff04d583f64efbb959eca86fd34969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ff04d583f64efbb959eca86fd34969" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32883 } }
I20260926 03:07:21.429168 10299 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71ff04d583f64efbb959eca86fd34969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:07:21.429246 10299 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71ff04d583f64efbb959eca86fd34969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ff04d583f64efbb959eca86fd34969, State: Initialized, Role: FOLLOWER
I20260926 03:07:21.429811 10299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71ff04d583f64efbb959eca86fd34969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ff04d583f64efbb959eca86fd34969" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32883 } }
I20260926 03:07:21.430050 10299 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71ff04d583f64efbb959eca86fd34969 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:07:21.430138 10299 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71ff04d583f64efbb959eca86fd34969 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:07:21.430258 10299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71ff04d583f64efbb959eca86fd34969 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:07:21.432408 10299 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71ff04d583f64efbb959eca86fd34969 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ff04d583f64efbb959eca86fd34969" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32883 } }
I20260926 03:07:21.432902 10299 leader_election.cc:302] T 00000000000000000000000000000000 P 71ff04d583f64efbb959eca86fd34969 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71ff04d583f64efbb959eca86fd34969; no voters:
I20260926 03:07:21.433501 10299 leader_election.cc:288] T 00000000000000000000000000000000 P 71ff04d583f64efbb959eca86fd34969 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:07:21.433621 10304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71ff04d583f64efbb959eca86fd34969 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:07:21.434307 10304 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71ff04d583f64efbb959eca86fd34969 [term 1 LEADER]: Becoming Leader. State: Replica: 71ff04d583f64efbb959eca86fd34969, State: Running, Role: LEADER
I20260926 03:07:21.435025 10304 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71ff04d583f64efbb959eca86fd34969 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ff04d583f64efbb959eca86fd34969" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32883 } }
I20260926 03:07:21.436189 10299 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71ff04d583f64efbb959eca86fd34969 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:07:21.437875 10306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71ff04d583f64efbb959eca86fd34969 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71ff04d583f64efbb959eca86fd34969" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ff04d583f64efbb959eca86fd34969" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32883 } } }
I20260926 03:07:21.438268 10306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71ff04d583f64efbb959eca86fd34969 [sys.catalog]: This master's current role is: LEADER
I20260926 03:07:21.438089 10305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71ff04d583f64efbb959eca86fd34969 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71ff04d583f64efbb959eca86fd34969. Latest consensus state: current_term: 1 leader_uuid: "71ff04d583f64efbb959eca86fd34969" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71ff04d583f64efbb959eca86fd34969" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32883 } } }
I20260926 03:07:21.438447 10305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71ff04d583f64efbb959eca86fd34969 [sys.catalog]: This master's current role is: LEADER
I20260926 03:07:21.440155 10311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:07:21.444700 10311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:07:21.453361 10311 catalog_manager.cc:1347] Generated new cluster ID: 712c0255564e4fa3b457e9a3506aa64f
I20260926 03:07:21.453485 10311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:07:21.494671 10311 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:07:21.495122 10311 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:07:21.500715 10311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71ff04d583f64efbb959eca86fd34969: Generated new TSK 0
I20260926 03:07:21.501657 10311 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:07:21.511358 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32883
--builtin_ntp_servers=127.0.97.20:41945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 03:07:21.861302 10323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:07:21.861589 10323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:07:21.861701 10323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:07:21.869048 10323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:07:21.869158 10323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:07:21.869302 10323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:07:21.878886 10323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:07:21.880203 10323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:07:21.882054 10323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:07:21.895627 10331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:21.895742 10329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:21.895576 10328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:07:21.896409 10323 server_base.cc:1034] running on GCE node
I20260926 03:07:21.897043 10323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:07:21.898113 10323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:07:21.899328 10323 hybrid_clock.cc:648] HybridClock initialized: now 1790392041899274 us; error 52 us; skew 500 ppm
I20260926 03:07:21.899670 10323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:07:21.902107 10323 webserver.cc:460] Webserver started at http://127.0.97.1:40967/ using document root <none> and password file <none>
I20260926 03:07:21.902856 10323 fs_manager.cc:359] Metadata directory not provided
I20260926 03:07:21.902957 10323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:07:21.903254 10323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:07:21.908262 10323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/data/instance:
uuid: "a314dacc33014a61be50075e4e090872"
format_stamp: "Formatted at 2026-09-26 03:07:21 on dist-test-slave-hmbw"
server_key: "4b38fcb14bf6c0063e46a4e843a6622d"
server_key_iv: "69d21ae2a3cae291e5d99606a7feaeee"
server_key_version: "encryptionkey@0"
I20260926 03:07:21.909101 10323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a314dacc33014a61be50075e4e090872"
format_stamp: "Formatted at 2026-09-26 03:07:21 on dist-test-slave-hmbw"
server_key: "4b38fcb14bf6c0063e46a4e843a6622d"
server_key_iv: "69d21ae2a3cae291e5d99606a7feaeee"
server_key_version: "encryptionkey@0"
I20260926 03:07:21.914752 10323 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 03:07:21.918789 10337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:21.920358 10323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:07:21.920564 10323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "a314dacc33014a61be50075e4e090872"
format_stamp: "Formatted at 2026-09-26 03:07:21 on dist-test-slave-hmbw"
server_key: "4b38fcb14bf6c0063e46a4e843a6622d"
server_key_iv: "69d21ae2a3cae291e5d99606a7feaeee"
server_key_version: "encryptionkey@0"
I20260926 03:07:21.920799 10323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:07:21.926852 10323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:07:21.927714 10323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:07:21.927995 10323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:07:21.928937 10323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:07:21.930330 10323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:07:21.930420 10323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:21.930583 10323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:07:21.930652 10323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:21.961623 10323 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40807
I20260926 03:07:21.961678 10449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40807 every 8 connection(s)
I20260926 03:07:21.963042 10323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:07:21.967389 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 10323
I20260926 03:07:21.967607 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:07:21.968161 388 external_mini_cluster.cc:1428] Setting key 6112d69b61dcea2c146c8ec2698c4807
I20260926 03:07:21.970976 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32883
--builtin_ntp_servers=127.0.97.20:41945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 03:07:21.971863 10452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:21.965667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58973 (local address 127.0.97.62:32883)
0926 03:07:21.966173 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:21.966178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:21.967431 (+ 1253us) server_negotiation.cc:407] Connection header received
0926 03:07:21.969397 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:21.969402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:21.969469 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:21.969565 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:21.971105 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:21.971111 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:21.971115 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:07:21.971135 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:21.971186 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:07:21.971200 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:07:21.971319 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:21.971498 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 03:07:21.971573 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":219,"threads_started":1}
I20260926 03:07:21.972854 10453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:21.966218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32883 (local address 127.0.97.1:58973)
0926 03:07:21.967261 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:21.967302 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:21.969092 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:21.969869 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:21.969883 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:21.970031 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:21.970789 (+ 758us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:07:21.970791 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:21.970831 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:07:21.970926 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:07:21.971283 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:07:21.971294 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:07:21.971303 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 03:07:21.971572 (+ 269us) client_negotiation.cc:240] Negotiation successful
0926 03:07:21.971853 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":361,"threads_started":1}
I20260926 03:07:21.974478 10450 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32883
I20260926 03:07:21.974805 10450 heartbeater.cc:464] Registering TS with master...
I20260926 03:07:21.975450 10450 heartbeater.cc:511] Master 127.0.97.62:32883 requested a full tablet report, sending...
I20260926 03:07:21.977679 10264 ts_manager.cc:194] Registered new tserver with Master: a314dacc33014a61be50075e4e090872 (127.0.97.1:40807)
W20260926 03:07:22.318086 10454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:07:22.318406 10454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:07:22.318547 10454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:07:22.326141 10454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:07:22.326265 10454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:07:22.326409 10454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:07:22.336774 10454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:07:22.338217 10454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:07:22.340281 10454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:07:22.353960 10459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:22.353984 10460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:22.353925 10462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:07:22.355232 10454 server_base.cc:1034] running on GCE node
I20260926 03:07:22.355840 10454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:07:22.356854 10454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:07:22.358129 10454 hybrid_clock.cc:648] HybridClock initialized: now 1790392042358056 us; error 72 us; skew 500 ppm
I20260926 03:07:22.358532 10454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:07:22.361157 10454 webserver.cc:460] Webserver started at http://127.0.97.2:32793/ using document root <none> and password file <none>
I20260926 03:07:22.361877 10454 fs_manager.cc:359] Metadata directory not provided
I20260926 03:07:22.361981 10454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:07:22.362282 10454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:07:22.365402 10454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/data/instance:
uuid: "0621f47ce86043adb425570e5e570aab"
format_stamp: "Formatted at 2026-09-26 03:07:22 on dist-test-slave-hmbw"
server_key: "accd6230bc12675b2e1485c985ab9ff9"
server_key_iv: "584e8600ae297a5257ad9c59b5c30ac7"
server_key_version: "encryptionkey@0"
I20260926 03:07:22.366238 10454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0621f47ce86043adb425570e5e570aab"
format_stamp: "Formatted at 2026-09-26 03:07:22 on dist-test-slave-hmbw"
server_key: "accd6230bc12675b2e1485c985ab9ff9"
server_key_iv: "584e8600ae297a5257ad9c59b5c30ac7"
server_key_version: "encryptionkey@0"
I20260926 03:07:22.371904 10454 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:07:22.376107 10468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:22.377616 10454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:07:22.377802 10454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "0621f47ce86043adb425570e5e570aab"
format_stamp: "Formatted at 2026-09-26 03:07:22 on dist-test-slave-hmbw"
server_key: "accd6230bc12675b2e1485c985ab9ff9"
server_key_iv: "584e8600ae297a5257ad9c59b5c30ac7"
server_key_version: "encryptionkey@0"
I20260926 03:07:22.378033 10454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:07:22.383850 10454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:07:22.384727 10454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:07:22.384984 10454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:07:22.385974 10454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:07:22.387475 10454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:07:22.387575 10454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:22.387679 10454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:07:22.387748 10454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:22.419586 10454 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40145
I20260926 03:07:22.419642 10580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40145 every 8 connection(s)
I20260926 03:07:22.420883 10454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:07:22.425287 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 10454
I20260926 03:07:22.425489 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:07:22.426074 388 external_mini_cluster.cc:1428] Setting key 86e7481a96384d71043eafe3af81b5d3
I20260926 03:07:22.429468 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32883
--builtin_ntp_servers=127.0.97.20:41945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 03:07:22.430814 10452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:22.423475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57605 (local address 127.0.97.62:32883)
0926 03:07:22.423680 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:22.423687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:22.425569 (+ 1882us) server_negotiation.cc:407] Connection header received
0926 03:07:22.427821 (+ 2252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:22.427826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:22.427983 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:22.428200 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:22.429977 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:22.429984 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:22.429988 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:07:22.430010 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:22.430051 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:07:22.430067 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:07:22.430206 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:22.430443 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 03:07:22.430530 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:07:22.431934 10583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:22.424042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32883 (local address 127.0.97.2:57605)
0926 03:07:22.425357 (+ 1315us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:22.425413 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:22.427418 (+ 2005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:22.428490 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:22.428504 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:22.428769 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:22.429582 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:07:22.429585 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:22.429635 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:07:22.429771 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:07:22.430228 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:07:22.430237 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:07:22.430245 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 03:07:22.430548 (+ 303us) client_negotiation.cc:240] Negotiation successful
0926 03:07:22.430834 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":298,"threads_started":1}
I20260926 03:07:22.433686 10581 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32883
I20260926 03:07:22.434172 10581 heartbeater.cc:464] Registering TS with master...
I20260926 03:07:22.434894 10581 heartbeater.cc:511] Master 127.0.97.62:32883 requested a full tablet report, sending...
I20260926 03:07:22.436498 10264 ts_manager.cc:194] Registered new tserver with Master: 0621f47ce86043adb425570e5e570aab (127.0.97.2:40145)
W20260926 03:07:22.765085 10584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:07:22.765369 10584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:07:22.765475 10584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:07:22.772892 10584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:07:22.773007 10584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:07:22.773147 10584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:07:22.782711 10584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:07:22.783988 10584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:07:22.785730 10584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:07:22.799026 10589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:22.799121 10592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:22.799091 10590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:07:22.800148 10584 server_base.cc:1034] running on GCE node
I20260926 03:07:22.800802 10584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:07:22.801774 10584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:07:22.802961 10584 hybrid_clock.cc:648] HybridClock initialized: now 1790392042802907 us; error 45 us; skew 500 ppm
I20260926 03:07:22.803270 10584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:07:22.805752 10584 webserver.cc:460] Webserver started at http://127.0.97.3:45843/ using document root <none> and password file <none>
I20260926 03:07:22.806454 10584 fs_manager.cc:359] Metadata directory not provided
I20260926 03:07:22.806593 10584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:07:22.806886 10584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:07:22.809885 10584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/data/instance:
uuid: "b94bdf5b3b09435c891eee48a9eea186"
format_stamp: "Formatted at 2026-09-26 03:07:22 on dist-test-slave-hmbw"
server_key: "23190f2dd6f92363929f21dc57d5bd38"
server_key_iv: "01388eed1d66f3717a33140f03de3093"
server_key_version: "encryptionkey@0"
I20260926 03:07:22.810712 10584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b94bdf5b3b09435c891eee48a9eea186"
format_stamp: "Formatted at 2026-09-26 03:07:22 on dist-test-slave-hmbw"
server_key: "23190f2dd6f92363929f21dc57d5bd38"
server_key_iv: "01388eed1d66f3717a33140f03de3093"
server_key_version: "encryptionkey@0"
I20260926 03:07:22.816129 10584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 03:07:22.819973 10598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:22.821406 10584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:07:22.821557 10584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "b94bdf5b3b09435c891eee48a9eea186"
format_stamp: "Formatted at 2026-09-26 03:07:22 on dist-test-slave-hmbw"
server_key: "23190f2dd6f92363929f21dc57d5bd38"
server_key_iv: "01388eed1d66f3717a33140f03de3093"
server_key_version: "encryptionkey@0"
I20260926 03:07:22.821787 10584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:07:22.827431 10584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:07:22.828214 10584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:07:22.828454 10584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:07:22.829357 10584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:07:22.830818 10584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:07:22.830901 10584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:22.830989 10584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:07:22.831063 10584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:22.861593 10584 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34535
I20260926 03:07:22.861646 10710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34535 every 8 connection(s)
I20260926 03:07:22.862960 10584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:07:22.870843 10452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:22.865415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36207 (local address 127.0.97.62:32883)
0926 03:07:22.865598 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:22.865603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:22.866974 (+ 1371us) server_negotiation.cc:407] Connection header received
0926 03:07:22.868635 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:22.868639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:22.868693 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:22.868779 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:22.870094 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:22.870098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:22.870102 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:07:22.870119 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:22.870152 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:07:22.870166 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:07:22.870274 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:22.870525 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 03:07:22.870606 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:07:22.871802 10713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:22.865892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32883 (local address 127.0.97.3:36207)
0926 03:07:22.866825 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:22.866866 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:22.868345 (+ 1479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:22.868972 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:22.868980 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:22.869076 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:22.869738 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:07:22.869741 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:22.869790 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:07:22.869882 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:07:22.870259 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:07:22.870266 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:07:22.870273 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 03:07:22.870568 (+ 295us) client_negotiation.cc:240] Negotiation successful
0926 03:07:22.870795 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":269,"threads_started":1}
I20260926 03:07:22.872267 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 10584
I20260926 03:07:22.872445 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391950058021-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:07:22.872823 388 external_mini_cluster.cc:1428] Setting key 09332507fcd30949b8b50bf67dff9712
I20260926 03:07:22.873144 10711 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32883
I20260926 03:07:22.873446 10711 heartbeater.cc:464] Registering TS with master...
I20260926 03:07:22.874179 10711 heartbeater.cc:511] Master 127.0.97.62:32883 requested a full tablet report, sending...
I20260926 03:07:22.875746 10264 ts_manager.cc:194] Registered new tserver with Master: b94bdf5b3b09435c891eee48a9eea186 (127.0.97.3:34535)
I20260926 03:07:22.888146 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 03:07:22.894176 10452 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:07:22.892672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57538 (local address 127.0.97.62:32883)
0926 03:07:22.892823 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:22.892827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:22.893307 (+ 480us) server_negotiation.cc:407] Connection header received
0926 03:07:22.893452 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:22.893456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:22.893512 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:22.893593 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:22.893970 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57538: BlockingRecv error: recv got EOF from 127.0.0.1:57538 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 03:07:22.894968 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 10323
I20260926 03:07:22.898208 10720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:22.895931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60450 (local address 127.0.97.1:40807)
0926 03:07:22.896447 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:22.896454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:22.896480 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:07:22.896538 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:22.896544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:22.896752 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:22.896879 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:22.897635 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:22.897642 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:22.897652 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:07:22.897668 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:22.897705 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:07:22.897718 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:07:22.897823 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:22.897898 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 03:07:22.897973 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":222,"threads_started":1}
I20260926 03:07:22.899547 10445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:07:22.979537 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 10323
I20260926 03:07:22.993045 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 10454
I20260926 03:07:22.996701 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:22.994069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48364 (local address 127.0.97.2:40145)
0926 03:07:22.994608 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:22.994621 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:22.994647 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:07:22.994706 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:22.994712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:22.994923 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:22.995058 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:22.995923 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:22.995930 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:22.995942 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:07:22.995961 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:22.996002 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:07:22.996017 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:07:22.996124 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:22.996310 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 03:07:22.996421 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":223,"threads_started":1}
I20260926 03:07:22.997923 10576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:07:23.079303 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 10454
I20260926 03:07:23.092100 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 10584
I20260926 03:07:23.095865 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:23.093087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47956 (local address 127.0.97.3:34535)
0926 03:07:23.093772 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:23.093779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:23.093810 (+ 31us) server_negotiation.cc:407] Connection header received
0926 03:07:23.093868 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:23.093874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:23.094107 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:23.094239 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:23.095266 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:23.095273 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:23.095284 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:07:23.095301 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:23.095337 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:07:23.095350 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:07:23.095483 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:23.095533 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 03:07:23.095610 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":398,"threads_started":1}
I20260926 03:07:23.097098 10706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:07:23.179917 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 10584
I20260926 03:07:23.192595 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 10233
I20260926 03:07:23.193754 10294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:07:23.295709 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 10233
2026-09-26T03:07:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2388 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:07:23 dist-test-slave-hmbw krb5kdc[10729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:07:23 dist-test-slave-hmbw krb5kdc[10729](info): set up 2 sockets
Sep 26 03:07:23 dist-test-slave-hmbw krb5kdc[10729](info): commencing operation
krb5kdc: starting...
W20260926 03:07:24.319368 388 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:07:24 dist-test-slave-hmbw krb5kdc[10729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:07:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:07:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:07:24.550911 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35103
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:07:24.873188 10742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:07:24.873481 10742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:07:24.873597 10742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:07:24.880843 10742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:07:24.880962 10742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:07:24.881019 10742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:07:24.881096 10742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:07:24.881160 10742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:07:24.891243 10742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:07:24.892563 10742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:07:24.893874 10742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:07:24.905107 10747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:24.905169 10750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:24.905135 10748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:07:24.905879 10742 server_base.cc:1034] running on GCE node
I20260926 03:07:24.906657 10742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:07:24.908126 10742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:07:24.909363 10742 hybrid_clock.cc:648] HybridClock initialized: now 1790392044909323 us; error 41 us; skew 500 ppm
I20260926 03:07:24.909701 10742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:07:24 dist-test-slave-hmbw krb5kdc[10729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392044, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:07:24.913054 10742 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:07:24.914710 10742 webserver.cc:460] Webserver started at http://127.0.97.62:37707/ using document root <none> and password file <none>
I20260926 03:07:24.915401 10742 fs_manager.cc:359] Metadata directory not provided
I20260926 03:07:24.915506 10742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:07:24.915758 10742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:07:24.918872 10742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "c8bc3a7b168c475493d4a00297f86176"
format_stamp: "Formatted at 2026-09-26 03:07:24 on dist-test-slave-hmbw"
server_key: "b6e0458ec80402f4fbb3a8d7bcc0b947"
server_key_iv: "685c34be9459c56989db9ca403124b54"
server_key_version: "encryptionkey@0"
I20260926 03:07:24.919612 10742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "c8bc3a7b168c475493d4a00297f86176"
format_stamp: "Formatted at 2026-09-26 03:07:24 on dist-test-slave-hmbw"
server_key: "b6e0458ec80402f4fbb3a8d7bcc0b947"
server_key_iv: "685c34be9459c56989db9ca403124b54"
server_key_version: "encryptionkey@0"
I20260926 03:07:24.925040 10742 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:07:24.928838 10757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:24.930460 10742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:07:24.930680 10742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "c8bc3a7b168c475493d4a00297f86176"
format_stamp: "Formatted at 2026-09-26 03:07:24 on dist-test-slave-hmbw"
server_key: "b6e0458ec80402f4fbb3a8d7bcc0b947"
server_key_iv: "685c34be9459c56989db9ca403124b54"
server_key_version: "encryptionkey@0"
I20260926 03:07:24.930873 10742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:07:24.948364 10742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:07:24.952572 10742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:07:24.952813 10742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:07:24.970013 10742 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35103
I20260926 03:07:24.970053 10808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35103 every 8 connection(s)
I20260926 03:07:24.971340 10742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:07:24.972831 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 10742
I20260926 03:07:24.973064 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:07:24.973852 388 external_mini_cluster.cc:1428] Setting key 9cca6fa4e22e28ded19982fd96ea936d
I20260926 03:07:24.975919 10809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 03:07:24 dist-test-slave-hmbw krb5kdc[10729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:07:24.989346 10809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8bc3a7b168c475493d4a00297f86176: Bootstrap starting.
I20260926 03:07:24.992751 10809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8bc3a7b168c475493d4a00297f86176: Neither blocks nor log segments found. Creating new log.
I20260926 03:07:24.994004 10809 log.cc:824] T 00000000000000000000000000000000 P c8bc3a7b168c475493d4a00297f86176: Log is configured to *not* fsync() on all Append() calls
I20260926 03:07:24.996829 10812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:24.975838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56524 (local address 127.0.97.62:35103)
0926 03:07:24.976557 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:24.976570 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:24.976610 (+ 40us) server_negotiation.cc:407] Connection header received
0926 03:07:24.978180 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:24.978201 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:24.978631 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:24.978964 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:24.980652 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:24.981964 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:24.983731 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:24.984454 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:24.984594 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:24.989442 (+ 4848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:24.989451 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:24.989469 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:07:24.989523 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:24.992563 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:07:24.993443 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:07:24.993451 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:07:24.993457 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:07:24.993642 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:07:24.994077 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:07:24.994081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:07:24.994084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:07:24.994806 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:07:24.995141 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:24.995426 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 03:07:24.995817 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":334,"threads_started":1}
I20260926 03:07:24.997071 10809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8bc3a7b168c475493d4a00297f86176: No bootstrap required, opened a new log
I20260926 03:07:25.004256 10809 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8bc3a7b168c475493d4a00297f86176 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8bc3a7b168c475493d4a00297f86176" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35103 } }
I20260926 03:07:25.004653 10809 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8bc3a7b168c475493d4a00297f86176 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:07:25.004748 10809 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8bc3a7b168c475493d4a00297f86176 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8bc3a7b168c475493d4a00297f86176, State: Initialized, Role: FOLLOWER
I20260926 03:07:25.005327 10809 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8bc3a7b168c475493d4a00297f86176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8bc3a7b168c475493d4a00297f86176" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35103 } }
I20260926 03:07:25.005558 10809 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8bc3a7b168c475493d4a00297f86176 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:07:25.005681 10809 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8bc3a7b168c475493d4a00297f86176 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:07:25.005831 10809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8bc3a7b168c475493d4a00297f86176 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:07:25.008154 10809 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8bc3a7b168c475493d4a00297f86176 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8bc3a7b168c475493d4a00297f86176" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35103 } }
I20260926 03:07:25.008708 10809 leader_election.cc:302] T 00000000000000000000000000000000 P c8bc3a7b168c475493d4a00297f86176 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8bc3a7b168c475493d4a00297f86176; no voters:
I20260926 03:07:25.009410 10809 leader_election.cc:288] T 00000000000000000000000000000000 P c8bc3a7b168c475493d4a00297f86176 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:07:25.009529 10814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8bc3a7b168c475493d4a00297f86176 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:07:25.010170 10814 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8bc3a7b168c475493d4a00297f86176 [term 1 LEADER]: Becoming Leader. State: Replica: c8bc3a7b168c475493d4a00297f86176, State: Running, Role: LEADER
I20260926 03:07:25.010764 10814 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8bc3a7b168c475493d4a00297f86176 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8bc3a7b168c475493d4a00297f86176" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35103 } }
I20260926 03:07:25.011510 10809 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8bc3a7b168c475493d4a00297f86176 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:07:25.013522 10815 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8bc3a7b168c475493d4a00297f86176 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8bc3a7b168c475493d4a00297f86176" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8bc3a7b168c475493d4a00297f86176" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35103 } } }
I20260926 03:07:25.013873 10815 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8bc3a7b168c475493d4a00297f86176 [sys.catalog]: This master's current role is: LEADER
I20260926 03:07:25.014802 10821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:07:25.014628 10816 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8bc3a7b168c475493d4a00297f86176 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8bc3a7b168c475493d4a00297f86176. Latest consensus state: current_term: 1 leader_uuid: "c8bc3a7b168c475493d4a00297f86176" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8bc3a7b168c475493d4a00297f86176" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35103 } } }
I20260926 03:07:25.015988 10816 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8bc3a7b168c475493d4a00297f86176 [sys.catalog]: This master's current role is: LEADER
I20260926 03:07:25.021718 10821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:07:25.033368 10821 catalog_manager.cc:1347] Generated new cluster ID: f30a7a22e77f48e3b8025d937f65be0c
I20260926 03:07:25.033512 10821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:07:25.052138 10821 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:07:25.053582 10821 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:07:25.067543 10821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8bc3a7b168c475493d4a00297f86176: Generated new TSK 0
I20260926 03:07:25.068437 10821 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:07:25.139334 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35103
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:07:25.467260 10835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:07:25.467564 10835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:07:25.467656 10835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:07:25.475095 10835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:07:25.475227 10835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:07:25.475378 10835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:07:25.485397 10835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:07:25.486760 10835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:07:25.488040 10835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:07:25.501467 10843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:25.501503 10841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:25.502281 10840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:07:25.504240 10835 server_base.cc:1034] running on GCE node
I20260926 03:07:25.504949 10835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:07:25.506000 10835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:07:25.507342 10835 hybrid_clock.cc:648] HybridClock initialized: now 1790392045507295 us; error 46 us; skew 500 ppm
I20260926 03:07:25.507730 10835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:07:25 dist-test-slave-hmbw krb5kdc[10729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:07:25.515955 10835 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:07:25.517465 10835 webserver.cc:460] Webserver started at http://127.0.97.1:34937/ using document root <none> and password file <none>
I20260926 03:07:25.518185 10835 fs_manager.cc:359] Metadata directory not provided
I20260926 03:07:25.518287 10835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:07:25.518586 10835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:07:25.521591 10835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-0/data/instance:
uuid: "20445391d8a745d9bf76ddee2d9112eb"
format_stamp: "Formatted at 2026-09-26 03:07:25 on dist-test-slave-hmbw"
server_key: "7b6268c0a48f26b7f3eb3ef7ecd513a8"
server_key_iv: "b9775f64e23e39003e923865d362885a"
server_key_version: "encryptionkey@0"
I20260926 03:07:25.522312 10835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-0/wal/instance:
uuid: "20445391d8a745d9bf76ddee2d9112eb"
format_stamp: "Formatted at 2026-09-26 03:07:25 on dist-test-slave-hmbw"
server_key: "7b6268c0a48f26b7f3eb3ef7ecd513a8"
server_key_iv: "b9775f64e23e39003e923865d362885a"
server_key_version: "encryptionkey@0"
I20260926 03:07:25.527670 10835 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260926 03:07:25.531276 10850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:25.532826 10835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:07:25.532984 10835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "20445391d8a745d9bf76ddee2d9112eb"
format_stamp: "Formatted at 2026-09-26 03:07:25 on dist-test-slave-hmbw"
server_key: "7b6268c0a48f26b7f3eb3ef7ecd513a8"
server_key_iv: "b9775f64e23e39003e923865d362885a"
server_key_version: "encryptionkey@0"
I20260926 03:07:25.533231 10835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:07:25.550009 10835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:07:25.554350 10835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:07:25.554628 10835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:07:25.555536 10835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:07:25.556852 10835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:07:25.556936 10835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:25.557036 10835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:07:25.557086 10835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:25.587721 10835 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41291
I20260926 03:07:25.587778 10962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41291 every 8 connection(s)
I20260926 03:07:25.588975 10835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:07:25.591266 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 10835
I20260926 03:07:25.591467 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:07:25.592197 388 external_mini_cluster.cc:1428] Setting key 514842ea8ea50c9dd9c114ddc6ff3982
Sep 26 03:07:25 dist-test-slave-hmbw krb5kdc[10729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:07:25.610284 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:25.591549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33305 (local address 127.0.97.62:35103)
0926 03:07:25.592076 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:25.592083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:25.593080 (+ 997us) server_negotiation.cc:407] Connection header received
0926 03:07:25.595001 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:25.595006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:25.595095 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:25.595230 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:25.597165 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:25.598083 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:25.599837 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:25.600399 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:25.600524 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:25.604445 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:25.604450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:25.604454 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:07:25.604489 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:25.607067 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:07:25.607994 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:07:25.607999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:07:25.608002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:07:25.608130 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:07:25.608606 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:07:25.608608 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:07:25.608610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:07:25.608861 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:07:25.608973 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:25.609801 (+ 828us) server_negotiation.cc:299] Negotiation successful
0926 03:07:25.609970 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":225,"threads_started":1}
I20260926 03:07:25.611840 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:25.592117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35103 (local address 127.0.97.1:33305)
0926 03:07:25.592914 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:25.592960 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:25.594744 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:25.595638 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:25.595653 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:25.596362 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:25.596983 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:25.597003 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:25.598276 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:25.598280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:25.599678 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:25.599691 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:25.600650 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:25.600655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:25.600887 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:25.601560 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:25.601584 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:25.604178 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:07:25.607244 (+ 3066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:07:25.607254 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:07:25.607274 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:07:25.607843 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:07:25.608232 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:07:25.608237 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:07:25.608241 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:07:25.608503 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:07:25.608966 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:07:25.608973 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:07:25.609413 (+ 440us) client_negotiation.cc:769] Sending connection context
0926 03:07:25.609727 (+ 314us) client_negotiation.cc:240] Negotiation successful
0926 03:07:25.610275 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":312,"threads_started":1}
I20260926 03:07:25.613646 10963 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35103
I20260926 03:07:25.613963 10963 heartbeater.cc:464] Registering TS with master...
I20260926 03:07:25.614854 10963 heartbeater.cc:511] Master 127.0.97.62:35103 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:07:25.617159 10774 ts_manager.cc:194] Registered new tserver with Master: 20445391d8a745d9bf76ddee2d9112eb (127.0.97.1:41291)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 03:07:25.618791 10774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:07:25.646732 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35103
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:07:25.982338 10969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:07:25.982714 10969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:07:25.982867 10969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:07:25.990646 10969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:07:25.990772 10969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:07:25.990921 10969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:07:26.001353 10969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:07:26.002790 10969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:07:26.004153 10969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:07:26.018342 10974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:26.018527 10977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:26.018530 10975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:07:26.019703 10969 server_base.cc:1034] running on GCE node
I20260926 03:07:26.020524 10969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:07:26.021588 10969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:07:26.022801 10969 hybrid_clock.cc:648] HybridClock initialized: now 1790392046022750 us; error 47 us; skew 500 ppm
I20260926 03:07:26.023124 10969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:07:26 dist-test-slave-hmbw krb5kdc[10729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:07:26.026762 10969 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:07:26.028553 10969 webserver.cc:460] Webserver started at http://127.0.97.2:44297/ using document root <none> and password file <none>
I20260926 03:07:26.029484 10969 fs_manager.cc:359] Metadata directory not provided
I20260926 03:07:26.029736 10969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:07:26.030180 10969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:07:26.033572 10969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-1/data/instance:
uuid: "d9b3ef14919047ddb28f958743ae5c3c"
format_stamp: "Formatted at 2026-09-26 03:07:26 on dist-test-slave-hmbw"
server_key: "1fb78d532e40bfe216985385b43b9f60"
server_key_iv: "480c30823eb992ffe4957eab912d509a"
server_key_version: "encryptionkey@0"
I20260926 03:07:26.034466 10969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d9b3ef14919047ddb28f958743ae5c3c"
format_stamp: "Formatted at 2026-09-26 03:07:26 on dist-test-slave-hmbw"
server_key: "1fb78d532e40bfe216985385b43b9f60"
server_key_iv: "480c30823eb992ffe4957eab912d509a"
server_key_version: "encryptionkey@0"
I20260926 03:07:26.040566 10969 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 03:07:26.044870 10984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:26.046934 10969 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 03:07:26.047142 10969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "d9b3ef14919047ddb28f958743ae5c3c"
format_stamp: "Formatted at 2026-09-26 03:07:26 on dist-test-slave-hmbw"
server_key: "1fb78d532e40bfe216985385b43b9f60"
server_key_iv: "480c30823eb992ffe4957eab912d509a"
server_key_version: "encryptionkey@0"
I20260926 03:07:26.047360 10969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:07:26.061849 10969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:07:26.066210 10969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:07:26.066536 10969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:07:26.067545 10969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:07:26.068974 10969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:07:26.069046 10969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:26.069128 10969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:07:26.069178 10969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:26.100546 10969 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35801
I20260926 03:07:26.100600 11096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35801 every 8 connection(s)
I20260926 03:07:26.101866 10969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:07:26.109850 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 10969
I20260926 03:07:26.110059 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:07:26.110612 388 external_mini_cluster.cc:1428] Setting key 359da779046a95c83cb279af9e11b54a
Sep 26 03:07:26 dist-test-slave-hmbw krb5kdc[10729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:07:26.123874 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:26.104601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40101 (local address 127.0.97.62:35103)
0926 03:07:26.104816 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:26.104823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:26.106098 (+ 1275us) server_negotiation.cc:407] Connection header received
0926 03:07:26.107971 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:26.107976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:26.108056 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:26.108182 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:26.109854 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.110952 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.113022 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.113651 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.113776 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:26.117626 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:26.117631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:26.117635 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:07:26.117672 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:26.120452 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:07:26.121362 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:07:26.121367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:07:26.121370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:07:26.121516 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:07:26.121957 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:07:26.121961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:07:26.121964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:07:26.122282 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:07:26.122422 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:26.123314 (+ 892us) server_negotiation.cc:299] Negotiation successful
0926 03:07:26.123539 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 03:07:26.124789 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:26.105101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35103 (local address 127.0.97.2:40101)
0926 03:07:26.105894 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:26.105935 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:26.107664 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:26.108375 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:26.108384 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:26.109015 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:26.109605 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:26.109623 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.111140 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.111144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:26.112859 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:26.112877 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.113892 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.113898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:26.114175 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:26.114883 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:26.114907 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:26.117310 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:07:26.120681 (+ 3371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:07:26.120690 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:07:26.120707 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:07:26.121238 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:07:26.121647 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:07:26.121651 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:07:26.121654 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:07:26.121848 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:07:26.122401 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:07:26.122408 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:07:26.122927 (+ 519us) client_negotiation.cc:769] Sending connection context
0926 03:07:26.123196 (+ 269us) client_negotiation.cc:240] Negotiation successful
0926 03:07:26.123616 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":318,"threads_started":1}
I20260926 03:07:26.126319 11097 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35103
I20260926 03:07:26.126663 11097 heartbeater.cc:464] Registering TS with master...
I20260926 03:07:26.127377 11097 heartbeater.cc:511] Master 127.0.97.62:35103 requested a full tablet report, sending...
I20260926 03:07:26.128962 10774 ts_manager.cc:194] Registered new tserver with Master: d9b3ef14919047ddb28f958743ae5c3c (127.0.97.2:35801)
I20260926 03:07:26.129824 10774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40101
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:07:26.162175 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35103
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:07:26.483443 11102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:07:26.483762 11102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:07:26.483897 11102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:07:26.491214 11102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:07:26.491357 11102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:07:26.491504 11102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:07:26.501562 11102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35103
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:07:26.502884 11102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:07:26.504213 11102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:07:26.519459 11107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:26.519521 11110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:07:26.519443 11108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:07:26.520195 11102 server_base.cc:1034] running on GCE node
I20260926 03:07:26.520932 11102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:07:26.521955 11102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:07:26.523226 11102 hybrid_clock.cc:648] HybridClock initialized: now 1790392046523145 us; error 77 us; skew 500 ppm
I20260926 03:07:26.523546 11102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:07:26 dist-test-slave-hmbw krb5kdc[10729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:07:26.526751 11102 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:07:26.528299 11102 webserver.cc:460] Webserver started at http://127.0.97.3:44767/ using document root <none> and password file <none>
I20260926 03:07:26.529011 11102 fs_manager.cc:359] Metadata directory not provided
I20260926 03:07:26.529119 11102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:07:26.529378 11102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:07:26.532353 11102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-2/data/instance:
uuid: "e81b02e62b454888bdf7c4d9278401c4"
format_stamp: "Formatted at 2026-09-26 03:07:26 on dist-test-slave-hmbw"
server_key: "2f1070f5ae5ab948988bf40383415e30"
server_key_iv: "dc002da372823e46195003cc900db8a1"
server_key_version: "encryptionkey@0"
I20260926 03:07:26.533116 11102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e81b02e62b454888bdf7c4d9278401c4"
format_stamp: "Formatted at 2026-09-26 03:07:26 on dist-test-slave-hmbw"
server_key: "2f1070f5ae5ab948988bf40383415e30"
server_key_iv: "dc002da372823e46195003cc900db8a1"
server_key_version: "encryptionkey@0"
I20260926 03:07:26.538686 11102 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 03:07:26.542336 11117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:26.543936 11102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:07:26.544164 11102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "e81b02e62b454888bdf7c4d9278401c4"
format_stamp: "Formatted at 2026-09-26 03:07:26 on dist-test-slave-hmbw"
server_key: "2f1070f5ae5ab948988bf40383415e30"
server_key_iv: "dc002da372823e46195003cc900db8a1"
server_key_version: "encryptionkey@0"
I20260926 03:07:26.544371 11102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:07:26.557615 11102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:07:26.561930 11102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:07:26.562213 11102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:07:26.563285 11102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:07:26.564630 11102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:07:26.564702 11102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:26.564798 11102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:07:26.564850 11102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:07:26.595621 11102 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37375
I20260926 03:07:26.595662 11229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37375 every 8 connection(s)
I20260926 03:07:26.596838 11102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:07:26.604017 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 11102
I20260926 03:07:26.604225 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:07:26.604766 388 external_mini_cluster.cc:1428] Setting key 053a5adf84709362b2a1de29a96b741a
Sep 26 03:07:26 dist-test-slave-hmbw krb5kdc[10729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:07:26.617736 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:26.599509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35605 (local address 127.0.97.62:35103)
0926 03:07:26.599691 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:26.599695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:26.600946 (+ 1251us) server_negotiation.cc:407] Connection header received
0926 03:07:26.602877 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:26.602882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:26.602965 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:26.603048 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:26.605045 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.605926 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.608289 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.608708 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.608819 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:26.612297 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:26.612302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:26.612305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:07:26.612348 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:26.614842 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:07:26.615630 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:07:26.615634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:07:26.615637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:07:26.615762 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:07:26.616165 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:07:26.616169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:07:26.616171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:07:26.616460 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:07:26.616586 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:26.617288 (+ 702us) server_negotiation.cc:299] Negotiation successful
0926 03:07:26.617457 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:07:26.618803 11232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:26.599946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35103 (local address 127.0.97.3:35605)
0926 03:07:26.600736 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:26.600782 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:26.602592 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:26.603301 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:26.603309 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:26.603961 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:07:26.604844 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:26.604873 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.606184 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.606190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:26.608094 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:26.608123 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.608905 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.608910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:26.609115 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:26.609807 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:07:26.609832 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:07:26.612090 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:07:26.615028 (+ 2938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:07:26.615038 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:07:26.615059 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:07:26.615476 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:07:26.615843 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:07:26.615846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:07:26.615848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:07:26.616033 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:07:26.616578 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:07:26.616584 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:07:26.616927 (+ 343us) client_negotiation.cc:769] Sending connection context
0926 03:07:26.617162 (+ 235us) client_negotiation.cc:240] Negotiation successful
0926 03:07:26.617558 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":332,"threads_started":1}
I20260926 03:07:26.620275 11230 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35103
I20260926 03:07:26.620626 11230 heartbeater.cc:464] Registering TS with master...
I20260926 03:07:26.621390 11230 heartbeater.cc:511] Master 127.0.97.62:35103 requested a full tablet report, sending...
I20260926 03:07:26.622164 10963 heartbeater.cc:503] Master 127.0.97.62:35103 was elected leader, sending a full tablet report...
I20260926 03:07:26.622923 10774 ts_manager.cc:194] Registered new tserver with Master: e81b02e62b454888bdf7c4d9278401c4 (127.0.97.3:37375)
I20260926 03:07:26.623891 10774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35605
I20260926 03:07:26.633612 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:07:26.653478 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:26.639990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56530 (local address 127.0.97.62:35103)
0926 03:07:26.640207 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:26.640214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:26.640765 (+ 551us) server_negotiation.cc:407] Connection header received
0926 03:07:26.640973 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:26.640977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:26.641056 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:26.641148 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:26.642993 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.643825 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.645280 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.645695 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.645806 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:26.648050 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:26.648055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:26.648058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:07:26.648114 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:26.650838 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:07:26.651534 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:07:26.651540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:07:26.651543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:07:26.651677 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:07:26.652077 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:07:26.652080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:07:26.652082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:07:26.652389 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:07:26.652577 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:26.652988 (+ 411us) server_negotiation.cc:299] Negotiation successful
0926 03:07:26.653184 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:07:26.657552 10774 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:07:26.664378 10774 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:07:26.697539 11242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:26.676796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37375 (local address 127.0.0.1:42630)
0926 03:07:26.684762 (+ 7966us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:26.684824 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:26.685050 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:26.687347 (+ 2297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:26.687356 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:26.687395 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:07:26.687831 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:26.687855 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.690835 (+ 2980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.690843 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:26.694628 (+ 3785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:26.694645 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.696450 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.696457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:26.696694 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:26.696774 (+ 80us) client_negotiation.cc:769] Sending connection context
0926 03:07:26.696964 (+ 190us) client_negotiation.cc:240] Negotiation successful
0926 03:07:26.697179 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7630,"thread_start_us":7443,"threads_started":1}
I20260926 03:07:26.697594 11244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:26.677266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42630 (local address 127.0.97.3:37375)
0926 03:07:26.678242 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:26.678252 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:26.685306 (+ 7054us) server_negotiation.cc:407] Connection header received
0926 03:07:26.685434 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:26.685444 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:26.685736 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:26.685944 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:07:26.688090 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.689833 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.694814 (+ 4981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.696049 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.696175 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:26.696415 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:26.697052 (+ 637us) server_negotiation.cc:299] Negotiation successful
0926 03:07:26.697231 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":729,"threads_started":1}
I20260926 03:07:26.703354 11245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:26.677540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33370 (local address 127.0.97.2:35801)
0926 03:07:26.695456 (+ 17916us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:26.695465 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:26.695490 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:07:26.695590 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:26.695598 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:26.695875 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:26.696100 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:07:26.697137 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.698861 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.701283 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.702273 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.702399 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:26.702635 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:26.702876 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 03:07:26.703059 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17778,"thread_start_us":17670,"threads_started":1}
I20260926 03:07:26.703397 11247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:26.677024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35801 (local address 127.0.0.1:33370)
0926 03:07:26.692926 (+ 15902us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:26.692956 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:26.693077 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:26.696105 (+ 3028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:26.696110 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:26.696138 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:07:26.696778 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:26.696795 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.699002 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.699008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:26.701042 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:26.701055 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.702517 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.702525 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:26.702690 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:26.702722 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 03:07:26.702808 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 03:07:26.702950 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15745,"mutex_wait_us":103,"thread_start_us":17846,"threads_started":1}
I20260926 03:07:26.702581 11165 tablet_service.cc:1467] Processing CreateTablet for tablet 6da58a6184fe4c48b3b283e6a5bcbb8f (DEFAULT_TABLE table=test-table [id=0eeb79dd19a84b91973356bd6a7a591d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:07:26.704109 11243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:26.678143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41291 (local address 127.0.0.1:52684)
0926 03:07:26.695143 (+ 17000us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:26.695168 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:26.695384 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:26.696357 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:26.696361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:26.696389 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:07:26.696942 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:26.696955 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.698898 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.698904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:26.701069 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:26.701085 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.703312 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.703319 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:26.703517 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:26.703551 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 03:07:26.703637 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 03:07:26.703767 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16880,"mutex_wait_us":25,"thread_start_us":14467,"threads_started":1}
I20260926 03:07:26.704414 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:26.677785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52684 (local address 127.0.97.1:41291)
0926 03:07:26.693592 (+ 15807us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:26.693603 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:26.695271 (+ 1668us) server_negotiation.cc:407] Connection header received
0926 03:07:26.695594 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:26.695600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:26.695875 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:26.696100 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:07:26.697159 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.698761 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.701381 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.702817 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.702946 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:26.703157 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:26.703919 (+ 762us) server_negotiation.cc:299] Negotiation successful
0926 03:07:26.704109 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15599,"thread_start_us":15405,"threads_started":1}
I20260926 03:07:26.704717 11165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6da58a6184fe4c48b3b283e6a5bcbb8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:07:26.707774 11032 tablet_service.cc:1467] Processing CreateTablet for tablet 6da58a6184fe4c48b3b283e6a5bcbb8f (DEFAULT_TABLE table=test-table [id=0eeb79dd19a84b91973356bd6a7a591d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:07:26.707849 10898 tablet_service.cc:1467] Processing CreateTablet for tablet 6da58a6184fe4c48b3b283e6a5bcbb8f (DEFAULT_TABLE table=test-table [id=0eeb79dd19a84b91973356bd6a7a591d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:07:26.709271 10898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6da58a6184fe4c48b3b283e6a5bcbb8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:07:26.709276 11032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6da58a6184fe4c48b3b283e6a5bcbb8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:07:26.718041 11248 tablet_bootstrap.cc:492] T 6da58a6184fe4c48b3b283e6a5bcbb8f P e81b02e62b454888bdf7c4d9278401c4: Bootstrap starting.
I20260926 03:07:26.721324 11248 tablet_bootstrap.cc:654] T 6da58a6184fe4c48b3b283e6a5bcbb8f P e81b02e62b454888bdf7c4d9278401c4: Neither blocks nor log segments found. Creating new log.
I20260926 03:07:26.722328 11248 log.cc:824] T 6da58a6184fe4c48b3b283e6a5bcbb8f P e81b02e62b454888bdf7c4d9278401c4: Log is configured to *not* fsync() on all Append() calls
I20260926 03:07:26.727306 11251 tablet_bootstrap.cc:492] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb: Bootstrap starting.
I20260926 03:07:26.731371 11251 tablet_bootstrap.cc:654] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb: Neither blocks nor log segments found. Creating new log.
I20260926 03:07:26.731557 11250 tablet_bootstrap.cc:492] T 6da58a6184fe4c48b3b283e6a5bcbb8f P d9b3ef14919047ddb28f958743ae5c3c: Bootstrap starting.
I20260926 03:07:26.732827 11251 log.cc:824] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb: Log is configured to *not* fsync() on all Append() calls
I20260926 03:07:26.733518 11248 tablet_bootstrap.cc:492] T 6da58a6184fe4c48b3b283e6a5bcbb8f P e81b02e62b454888bdf7c4d9278401c4: No bootstrap required, opened a new log
I20260926 03:07:26.733803 11248 ts_tablet_manager.cc:1397] T 6da58a6184fe4c48b3b283e6a5bcbb8f P e81b02e62b454888bdf7c4d9278401c4: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20260926 03:07:26.735764 11250 tablet_bootstrap.cc:654] T 6da58a6184fe4c48b3b283e6a5bcbb8f P d9b3ef14919047ddb28f958743ae5c3c: Neither blocks nor log segments found. Creating new log.
I20260926 03:07:26.737282 11250 log.cc:824] T 6da58a6184fe4c48b3b283e6a5bcbb8f P d9b3ef14919047ddb28f958743ae5c3c: Log is configured to *not* fsync() on all Append() calls
I20260926 03:07:26.742064 11248 raft_consensus.cc:348] T 6da58a6184fe4c48b3b283e6a5bcbb8f P e81b02e62b454888bdf7c4d9278401c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81b02e62b454888bdf7c4d9278401c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37375 } } peers { permanent_uuid: "d9b3ef14919047ddb28f958743ae5c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35801 } } peers { permanent_uuid: "20445391d8a745d9bf76ddee2d9112eb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41291 } }
I20260926 03:07:26.742393 11251 tablet_bootstrap.cc:492] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb: No bootstrap required, opened a new log
I20260926 03:07:26.742430 11248 raft_consensus.cc:374] T 6da58a6184fe4c48b3b283e6a5bcbb8f P e81b02e62b454888bdf7c4d9278401c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:07:26.742547 11248 raft_consensus.cc:729] T 6da58a6184fe4c48b3b283e6a5bcbb8f P e81b02e62b454888bdf7c4d9278401c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e81b02e62b454888bdf7c4d9278401c4, State: Initialized, Role: FOLLOWER
I20260926 03:07:26.742743 11251 ts_tablet_manager.cc:1397] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.006s
I20260926 03:07:26.742964 11250 tablet_bootstrap.cc:492] T 6da58a6184fe4c48b3b283e6a5bcbb8f P d9b3ef14919047ddb28f958743ae5c3c: No bootstrap required, opened a new log
I20260926 03:07:26.743224 11250 ts_tablet_manager.cc:1397] T 6da58a6184fe4c48b3b283e6a5bcbb8f P d9b3ef14919047ddb28f958743ae5c3c: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260926 03:07:26.743151 11248 consensus_queue.cc:260] T 6da58a6184fe4c48b3b283e6a5bcbb8f P e81b02e62b454888bdf7c4d9278401c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81b02e62b454888bdf7c4d9278401c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37375 } } peers { permanent_uuid: "d9b3ef14919047ddb28f958743ae5c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35801 } } peers { permanent_uuid: "20445391d8a745d9bf76ddee2d9112eb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41291 } }
I20260926 03:07:26.744279 11230 heartbeater.cc:503] Master 127.0.97.62:35103 was elected leader, sending a full tablet report...
I20260926 03:07:26.744799 11248 ts_tablet_manager.cc:1428] T 6da58a6184fe4c48b3b283e6a5bcbb8f P e81b02e62b454888bdf7c4d9278401c4: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260926 03:07:26.753569 11250 raft_consensus.cc:348] T 6da58a6184fe4c48b3b283e6a5bcbb8f P d9b3ef14919047ddb28f958743ae5c3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81b02e62b454888bdf7c4d9278401c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37375 } } peers { permanent_uuid: "d9b3ef14919047ddb28f958743ae5c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35801 } } peers { permanent_uuid: "20445391d8a745d9bf76ddee2d9112eb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41291 } }
I20260926 03:07:26.754135 11250 raft_consensus.cc:374] T 6da58a6184fe4c48b3b283e6a5bcbb8f P d9b3ef14919047ddb28f958743ae5c3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:07:26.754235 11250 raft_consensus.cc:729] T 6da58a6184fe4c48b3b283e6a5bcbb8f P d9b3ef14919047ddb28f958743ae5c3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9b3ef14919047ddb28f958743ae5c3c, State: Initialized, Role: FOLLOWER
I20260926 03:07:26.754024 11251 raft_consensus.cc:348] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81b02e62b454888bdf7c4d9278401c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37375 } } peers { permanent_uuid: "d9b3ef14919047ddb28f958743ae5c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35801 } } peers { permanent_uuid: "20445391d8a745d9bf76ddee2d9112eb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41291 } }
I20260926 03:07:26.754431 11251 raft_consensus.cc:374] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:07:26.754546 11251 raft_consensus.cc:729] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20445391d8a745d9bf76ddee2d9112eb, State: Initialized, Role: FOLLOWER
I20260926 03:07:26.754985 11250 consensus_queue.cc:260] T 6da58a6184fe4c48b3b283e6a5bcbb8f P d9b3ef14919047ddb28f958743ae5c3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81b02e62b454888bdf7c4d9278401c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37375 } } peers { permanent_uuid: "d9b3ef14919047ddb28f958743ae5c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35801 } } peers { permanent_uuid: "20445391d8a745d9bf76ddee2d9112eb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41291 } }
I20260926 03:07:26.755139 11251 consensus_queue.cc:260] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81b02e62b454888bdf7c4d9278401c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37375 } } peers { permanent_uuid: "d9b3ef14919047ddb28f958743ae5c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35801 } } peers { permanent_uuid: "20445391d8a745d9bf76ddee2d9112eb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41291 } }
I20260926 03:07:26.756217 11097 heartbeater.cc:503] Master 127.0.97.62:35103 was elected leader, sending a full tablet report...
I20260926 03:07:26.756865 11250 ts_tablet_manager.cc:1428] T 6da58a6184fe4c48b3b283e6a5bcbb8f P d9b3ef14919047ddb28f958743ae5c3c: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260926 03:07:26.757956 11251 ts_tablet_manager.cc:1428] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb: Time spent starting tablet: real 0.015s user 0.005s sys 0.007s
I20260926 03:07:26.759558 11256 raft_consensus.cc:482] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:07:26.759783 11256 raft_consensus.cc:504] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81b02e62b454888bdf7c4d9278401c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37375 } } peers { permanent_uuid: "d9b3ef14919047ddb28f958743ae5c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35801 } } peers { permanent_uuid: "20445391d8a745d9bf76ddee2d9112eb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41291 } }
I20260926 03:07:26.762595 11256 leader_election.cc:288] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e81b02e62b454888bdf7c4d9278401c4 (127.0.97.3:37375), d9b3ef14919047ddb28f958743ae5c3c (127.0.97.2:35801)
I20260926 03:07:26.769095 11244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:26.762198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36689 (local address 127.0.97.3:37375)
0926 03:07:26.762399 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:26.762404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:26.762450 (+ 46us) server_negotiation.cc:407] Connection header received
0926 03:07:26.762712 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:26.762717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:26.762797 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:26.762880 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:07:26.763589 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.764546 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.767080 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.768226 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.768383 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:26.768493 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:26.768758 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 03:07:26.768884 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:07:26.769349 11257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:26.761802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37375 (local address 127.0.97.1:36689)
0926 03:07:26.762368 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:26.762387 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:26.762555 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:26.762934 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:26.762939 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:26.762976 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:07:26.763418 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:26.763431 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.764694 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.764698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:26.766808 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:26.766823 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.768461 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.768467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:26.768596 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:26.768622 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 03:07:26.768717 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 03:07:26.768873 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":23,"thread_start_us":288,"threads_started":1}
I20260926 03:07:26.769546 11245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:26.763049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38715 (local address 127.0.97.2:35801)
0926 03:07:26.763251 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:26.763258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:26.763736 (+ 478us) server_negotiation.cc:407] Connection header received
0926 03:07:26.763835 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:26.763840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:26.763912 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:26.764009 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:07:26.764416 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.765168 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.767390 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.768610 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.768725 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:26.768810 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:26.769173 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 03:07:26.769301 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:07:26.769814 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:26.762938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35801 (local address 127.0.97.1:38715)
0926 03:07:26.763647 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0926 03:07:26.763663 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:07:26.763748 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:07:26.764009 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:07:26.764012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:07:26.764026 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:07:26.764280 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:26.764289 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.765308 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.765313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:26.767237 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:07:26.767246 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.768827 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.768833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:07:26.768979 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:26.769018 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 03:07:26.769116 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 03:07:26.769483 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":4,"thread_start_us":400,"threads_started":1}
I20260926 03:07:26.770395 11185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6da58a6184fe4c48b3b283e6a5bcbb8f" candidate_uuid: "20445391d8a745d9bf76ddee2d9112eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e81b02e62b454888bdf7c4d9278401c4" is_pre_election: true
I20260926 03:07:26.771059 11185 raft_consensus.cc:2454] T 6da58a6184fe4c48b3b283e6a5bcbb8f P e81b02e62b454888bdf7c4d9278401c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20445391d8a745d9bf76ddee2d9112eb in term 0.
I20260926 03:07:26.770876 11052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6da58a6184fe4c48b3b283e6a5bcbb8f" candidate_uuid: "20445391d8a745d9bf76ddee2d9112eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9b3ef14919047ddb28f958743ae5c3c" is_pre_election: true
I20260926 03:07:26.771435 11052 raft_consensus.cc:2454] T 6da58a6184fe4c48b3b283e6a5bcbb8f P d9b3ef14919047ddb28f958743ae5c3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20445391d8a745d9bf76ddee2d9112eb in term 0.
I20260926 03:07:26.771787 10853 leader_election.cc:302] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20445391d8a745d9bf76ddee2d9112eb, e81b02e62b454888bdf7c4d9278401c4; no voters:
I20260926 03:07:26.772217 11256 raft_consensus.cc:2781] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:07:26.772349 11256 raft_consensus.cc:482] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:07:26.772439 11256 raft_consensus.cc:3037] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:07:26.774591 11256 raft_consensus.cc:504] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81b02e62b454888bdf7c4d9278401c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37375 } } peers { permanent_uuid: "d9b3ef14919047ddb28f958743ae5c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35801 } } peers { permanent_uuid: "20445391d8a745d9bf76ddee2d9112eb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41291 } }
I20260926 03:07:26.775167 11256 leader_election.cc:288] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb [CANDIDATE]: Term 1 election: Requested vote from peers e81b02e62b454888bdf7c4d9278401c4 (127.0.97.3:37375), d9b3ef14919047ddb28f958743ae5c3c (127.0.97.2:35801)
I20260926 03:07:26.775743 11052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6da58a6184fe4c48b3b283e6a5bcbb8f" candidate_uuid: "20445391d8a745d9bf76ddee2d9112eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9b3ef14919047ddb28f958743ae5c3c"
I20260926 03:07:26.775743 11185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6da58a6184fe4c48b3b283e6a5bcbb8f" candidate_uuid: "20445391d8a745d9bf76ddee2d9112eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e81b02e62b454888bdf7c4d9278401c4"
I20260926 03:07:26.775982 11185 raft_consensus.cc:3037] T 6da58a6184fe4c48b3b283e6a5bcbb8f P e81b02e62b454888bdf7c4d9278401c4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:07:26.775959 11052 raft_consensus.cc:3037] T 6da58a6184fe4c48b3b283e6a5bcbb8f P d9b3ef14919047ddb28f958743ae5c3c [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:07:26.779398 11185 raft_consensus.cc:2454] T 6da58a6184fe4c48b3b283e6a5bcbb8f P e81b02e62b454888bdf7c4d9278401c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20445391d8a745d9bf76ddee2d9112eb in term 1.
I20260926 03:07:26.779398 11052 raft_consensus.cc:2454] T 6da58a6184fe4c48b3b283e6a5bcbb8f P d9b3ef14919047ddb28f958743ae5c3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20445391d8a745d9bf76ddee2d9112eb in term 1.
I20260926 03:07:26.779937 10853 leader_election.cc:302] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 20445391d8a745d9bf76ddee2d9112eb, d9b3ef14919047ddb28f958743ae5c3c, e81b02e62b454888bdf7c4d9278401c4; no voters:
I20260926 03:07:26.780326 11256 raft_consensus.cc:2781] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:07:26.780802 11256 raft_consensus.cc:686] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb [term 1 LEADER]: Becoming Leader. State: Replica: 20445391d8a745d9bf76ddee2d9112eb, State: Running, Role: LEADER
I20260926 03:07:26.781266 11256 consensus_queue.cc:237] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81b02e62b454888bdf7c4d9278401c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37375 } } peers { permanent_uuid: "d9b3ef14919047ddb28f958743ae5c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35801 } } peers { permanent_uuid: "20445391d8a745d9bf76ddee2d9112eb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41291 } }
I20260926 03:07:26.786247 10773 catalog_manager.cc:5495] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 20445391d8a745d9bf76ddee2d9112eb (127.0.97.1). New cstate: current_term: 1 leader_uuid: "20445391d8a745d9bf76ddee2d9112eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e81b02e62b454888bdf7c4d9278401c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9b3ef14919047ddb28f958743ae5c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20445391d8a745d9bf76ddee2d9112eb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41291 } health_report { overall_health: HEALTHY } } }
I20260926 03:07:26.814980 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:26.809168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52686 (local address 127.0.97.1:41291)
0926 03:07:26.809385 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:26.809391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:26.809457 (+ 66us) server_negotiation.cc:407] Connection header received
0926 03:07:26.809693 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:26.809698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:26.809792 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:26.809924 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:07:26.810308 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.811088 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.812602 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.813145 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.813275 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:26.813822 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:07:26.814351 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:07:26.814530 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:26.814630 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 03:07:26.814718 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 03:07:26.823777 11052 raft_consensus.cc:1261] T 6da58a6184fe4c48b3b283e6a5bcbb8f P d9b3ef14919047ddb28f958743ae5c3c [term 1 FOLLOWER]: Refusing update from remote peer 20445391d8a745d9bf76ddee2d9112eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:07:26.823777 11185 raft_consensus.cc:1261] T 6da58a6184fe4c48b3b283e6a5bcbb8f P e81b02e62b454888bdf7c4d9278401c4 [term 1 FOLLOWER]: Refusing update from remote peer 20445391d8a745d9bf76ddee2d9112eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:07:26.825057 11259 consensus_queue.cc:1035] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e81b02e62b454888bdf7c4d9278401c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:07:26.825505 11256 consensus_queue.cc:1035] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9b3ef14919047ddb28f958743ae5c3c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:07:26.854565 10772 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:07:26.854909 10772 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:07:26.859031 10772 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8bc3a7b168c475493d4a00297f86176: Sending DeleteTablet for 3 replicas of tablet 6da58a6184fe4c48b3b283e6a5bcbb8f
I20260926 03:07:26.860280 10898 tablet_service.cc:1514] Processing DeleteTablet for tablet 6da58a6184fe4c48b3b283e6a5bcbb8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:07:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52684
I20260926 03:07:26.860260 11032 tablet_service.cc:1514] Processing DeleteTablet for tablet 6da58a6184fe4c48b3b283e6a5bcbb8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:07:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33370
I20260926 03:07:26.860252 11165 tablet_service.cc:1514] Processing DeleteTablet for tablet 6da58a6184fe4c48b3b283e6a5bcbb8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:07:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42630
I20260926 03:07:26.864136 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 10835
I20260926 03:07:26.873533 11274 tablet_replica.cc:330] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb: stopping tablet replica
I20260926 03:07:26.874132 11274 raft_consensus.cc:2229] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:07:26.874938 11274 raft_consensus.cc:2258] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:07:26.875716 11272 tablet_replica.cc:330] T 6da58a6184fe4c48b3b283e6a5bcbb8f P d9b3ef14919047ddb28f958743ae5c3c: stopping tablet replica
I20260926 03:07:26.876076 11273 tablet_replica.cc:330] T 6da58a6184fe4c48b3b283e6a5bcbb8f P e81b02e62b454888bdf7c4d9278401c4: stopping tablet replica
I20260926 03:07:26.876154 11272 raft_consensus.cc:2229] T 6da58a6184fe4c48b3b283e6a5bcbb8f P d9b3ef14919047ddb28f958743ae5c3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:07:26.876570 11273 raft_consensus.cc:2229] T 6da58a6184fe4c48b3b283e6a5bcbb8f P e81b02e62b454888bdf7c4d9278401c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:07:26.876675 11272 raft_consensus.cc:2258] T 6da58a6184fe4c48b3b283e6a5bcbb8f P d9b3ef14919047ddb28f958743ae5c3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:07:26.877116 11273 raft_consensus.cc:2258] T 6da58a6184fe4c48b3b283e6a5bcbb8f P e81b02e62b454888bdf7c4d9278401c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:07:26.878963 11274 ts_tablet_manager.cc:1905] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:07:26.878960 11272 ts_tablet_manager.cc:1905] T 6da58a6184fe4c48b3b283e6a5bcbb8f P d9b3ef14919047ddb28f958743ae5c3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:07:26.879274 11273 ts_tablet_manager.cc:1905] T 6da58a6184fe4c48b3b283e6a5bcbb8f P e81b02e62b454888bdf7c4d9278401c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:07:26.885982 11274 ts_tablet_manager.cc:1918] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:07:26.885941 11272 ts_tablet_manager.cc:1918] T 6da58a6184fe4c48b3b283e6a5bcbb8f P d9b3ef14919047ddb28f958743ae5c3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:07:26.886142 11272 log.cc:1196] T 6da58a6184fe4c48b3b283e6a5bcbb8f P d9b3ef14919047ddb28f958743ae5c3c: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-1/wal/wals/6da58a6184fe4c48b3b283e6a5bcbb8f
I20260926 03:07:26.886335 11274 log.cc:1196] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-0/wal/wals/6da58a6184fe4c48b3b283e6a5bcbb8f
I20260926 03:07:26.886677 11272 ts_tablet_manager.cc:1939] T 6da58a6184fe4c48b3b283e6a5bcbb8f P d9b3ef14919047ddb28f958743ae5c3c: Deleting consensus metadata
I20260926 03:07:26.887039 11274 ts_tablet_manager.cc:1939] T 6da58a6184fe4c48b3b283e6a5bcbb8f P 20445391d8a745d9bf76ddee2d9112eb: Deleting consensus metadata
I20260926 03:07:26.888736 10758 catalog_manager.cc:4841] TS 20445391d8a745d9bf76ddee2d9112eb (127.0.97.1:41291): tablet 6da58a6184fe4c48b3b283e6a5bcbb8f (table test-table [id=0eeb79dd19a84b91973356bd6a7a591d]) successfully deleted
I20260926 03:07:26.889046 10761 catalog_manager.cc:4841] TS d9b3ef14919047ddb28f958743ae5c3c (127.0.97.2:35801): tablet 6da58a6184fe4c48b3b283e6a5bcbb8f (table test-table [id=0eeb79dd19a84b91973356bd6a7a591d]) successfully deleted
I20260926 03:07:26.891407 11273 ts_tablet_manager.cc:1918] T 6da58a6184fe4c48b3b283e6a5bcbb8f P e81b02e62b454888bdf7c4d9278401c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:07:26.891590 11273 log.cc:1196] T 6da58a6184fe4c48b3b283e6a5bcbb8f P e81b02e62b454888bdf7c4d9278401c4: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391950058021-388-0/minicluster-data/ts-2/wal/wals/6da58a6184fe4c48b3b283e6a5bcbb8f
I20260926 03:07:26.892128 11273 ts_tablet_manager.cc:1939] T 6da58a6184fe4c48b3b283e6a5bcbb8f P e81b02e62b454888bdf7c4d9278401c4: Deleting consensus metadata
I20260926 03:07:26.893937 10760 catalog_manager.cc:4841] TS e81b02e62b454888bdf7c4d9278401c4 (127.0.97.3:37375): tablet 6da58a6184fe4c48b3b283e6a5bcbb8f (table test-table [id=0eeb79dd19a84b91973356bd6a7a591d]) successfully deleted
Sep 26 03:07:26 dist-test-slave-hmbw krb5kdc[10729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:07:26.904229 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:26.865609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52688 (local address 127.0.97.1:41291)
0926 03:07:26.865791 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:26.865797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:26.884933 (+ 19136us) server_negotiation.cc:407] Connection header received
0926 03:07:26.885444 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:26.885451 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:26.885535 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:26.885625 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:26.890718 (+ 5093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.891672 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.893017 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:26.893459 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:26.893570 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:26.897803 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:26.897829 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:26.897851 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:07:26.897911 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:26.901086 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:07:26.902021 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:07:26.902029 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:07:26.902032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:07:26.902190 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:07:26.902593 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:07:26.902596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:07:26.902598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:07:26.903030 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:07:26.903267 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:26.903580 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 03:07:26.903819 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:07:26.905010 10958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:07:27.003993 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 10835
I20260926 03:07:27.019893 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 10969
Sep 26 03:07:27 dist-test-slave-hmbw krb5kdc[10729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:07:27.034973 11245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:27.020652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33378 (local address 127.0.97.2:35801)
0926 03:07:27.020800 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:27.020805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:27.020823 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:07:27.020929 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:27.020933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:27.021006 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:27.021109 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:27.022345 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:27.023164 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:27.024486 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:27.024909 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:27.025016 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:27.029302 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:27.029318 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:27.029335 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:07:27.029380 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:27.032189 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:07:27.032899 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:07:27.032906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:07:27.032908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:07:27.033115 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:07:27.033558 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:07:27.033563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:07:27.033567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:07:27.033959 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:07:27.034190 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:27.034451 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 03:07:27.034719 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 03:07:27.035743 11092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:07:27.124064 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 10969
I20260926 03:07:27.139799 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 11102
Sep 26 03:07:27 dist-test-slave-hmbw krb5kdc[10729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:07:27.155553 11244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:07:27.140597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42644 (local address 127.0.97.3:37375)
0926 03:07:27.140774 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 03:07:27.140779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:07:27.140800 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:07:27.140937 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:07:27.140943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:07:27.141014 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:07:27.141097 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:07:27.142467 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:27.143223 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:27.144397 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:07:27.144834 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:07:27.144945 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:07:27.149344 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:07:27.149365 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:07:27.149382 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:07:27.149427 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:07:27.152283 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:07:27.153122 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:07:27.153137 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:07:27.153139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:07:27.153310 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:07:27.153799 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:07:27.153803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:07:27.153805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:07:27.154196 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:07:27.154464 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 03:07:27.154989 (+ 525us) server_negotiation.cc:299] Negotiation successful
0926 03:07:27.155268 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 03:07:27.156267 11225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:07:27.249562 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 11102
I20260926 03:07:27.263525 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 10742
I20260926 03:07:27.264689 10804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:07:27.371848 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 10742
2026-09-26T03:07:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4075 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:07:27 dist-test-slave-hmbw krb5kdc[11282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:07:27 dist-test-slave-hmbw krb5kdc[11282](info): set up 2 sockets
Sep 26 03:07:27 dist-test-slave-hmbw krb5kdc[11282](info): commencing operation
krb5kdc: starting...
W20260926 03:07:28.395197 388 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:07:28 dist-test-slave-hmbw krb5kdc[11282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:07:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:07:28Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260926 03:07:28.722829 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskuHAswW/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/postgres -l logfile start
2026-09-26 03:07:31.544 UTC [11312] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 03:07:31.545 UTC [11312] LOG: listening on IPv4 address "127.0.97.20", port 53695
2026-09-26 03:07:31.549 UTC [11312] LOG: listening on Unix socket "/tmp/.s.PGSQL.53695"
2026-09-26 03:07:31.571 UTC [11315] LOG: database system was shut down at 2026-09-26 03:07:29 UTC
2026-09-26 03:07:31.576 UTC [11312] LOG: database system is ready to accept connections
I20260926 03:07:32.494920 388 mini_postgres.cc:96] Postgres bound to 53695
2026-09-26 03:07:32.512 UTC [11323] FATAL: database "slave" does not exist
127.0.97.20:53695 - accepting connections
I20260926 03:07:32.513186 388 mini_ranger.cc:161] Starting Ranger...
I20260926 03:07:32.540371 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 03:07:33.139842 388 mini_ranger.cc:87] Created ranger Postgres database
I20260926 03:07:33.139973 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ranger-admin
2026-09-26 03:07:33,437 [I] DB FLAVOR :POSTGRES
2026-09-26 03:07:33,438 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 03:07:33,438 [I] Checking connection..
2026-09-26 03:07:33,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:53695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 03:07:33,715 [I] Checking connection passed.
2026-09-26 03:07:33,715 [I] --------- Verifying version history table ---------
2026-09-26 03:07:33,715 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:53695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:07:33,955 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 03:07:33,955 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:53695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:07:34,224 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 03:07:34,224 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 03:07:34,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:53695/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 03:07:34.469 UTC [11387] WARNING: there is no transaction in progress
2026-09-26 03:07:34,474 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 03:07:34,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:53695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:07:34,724 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 03:07:34,724 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 03:07:34,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:53695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 03:07:34,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:53695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 03:07:35,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:53695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hmbw.c.gcp-upstream.internal','N') ;"
2026-09-26 03:07:35,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:53695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 03:07:35,736 [I] Table x_portal_user does not exist in database ranger
2026-09-26 03:07:35,737 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:53695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 03:07:36,002 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 03:07:36,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:53695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 03:07:36,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:53695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 03:07:36,487 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 03:07:36,487 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:53695/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 03:07:37.519 UTC [11495] WARNING: there is no transaction in progress
2026-09-26 03:07:37.537 UTC [11495] WARNING: there is no transaction in progress
2026-09-26 03:07:37.554 UTC [11495] WARNING: there is no transaction in progress
2026-09-26 03:07:37.573 UTC [11495] WARNING: there is no transaction in progress
2026-09-26 03:07:37.591 UTC [11495] WARNING: there is no transaction in progress
2026-09-26 03:07:37.610 UTC [11495] WARNING: there is no transaction in progress
2026-09-26 03:07:37.671 UTC [11495] WARNING: there is no transaction in progress
2026-09-26 03:07:37.679 UTC [11495] WARNING: there is no transaction in progress
2026-09-26 03:07:37.695 UTC [11495] WARNING: there is no transaction in progress
2026-09-26 03:07:37.706 UTC [11495] WARNING: there is no transaction in progress
2026-09-26 03:07:37.718 UTC [11495] WARNING: there is no transaction in progress
2026-09-26 03:07:37.732 UTC [11495] WARNING: there is no transaction in progress
2026-09-26 03:07:37.742 UTC [11495] WARNING: there is no transaction in progress
2026-09-26 03:07:37.750 UTC [11495] WARNING: there is no transaction in progress
2026-09-26 03:07:37.757 UTC [11495] WARNING: there is no transaction in progress
2026-09-26 03:07:37.765 UTC [11495] WARNING: there is no transaction in progress
2026-09-26 03:07:38,485 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 03:07:38,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:53695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 03:07:38,739 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 03:07:38,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:53695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 03:07:38,973 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 03:07:38,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:53695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 03:07:39,210 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 03:07:39,210 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:53695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 03:07:39,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:53695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 03:07:39,735 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:53695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hmbw.c.gcp-upstream.internal';"
2026-09-26 03:07:39,978 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 03:07:39,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:53695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 03:07:40,255 [I] DB_PATCHES have already been applied
I20260926 03:07:40.260924 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskuHAswW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 03:07:40.261013 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260926 03:07:40.276171 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:55769
Sep 26, 2026 3:07:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:07:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:07:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 3:07:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 3:07:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 3:07:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 3:07:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 03:07:41.318991 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.806s user 0.000s sys 0.042s
Sep 26, 2026 3:08:01 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 3:08:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 3:08:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 3:08:01 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 3:08:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 3:08:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 3:08:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 3:08:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 3:08:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 3:08:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 3:08:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 3:08:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 3:08:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 3:08:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 3:08:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 3:08:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 3:08:01 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 3:08:02 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 03:08:03.177838 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 03:08:03 dist-test-slave-hmbw krb5kdc[11282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392083, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 03:08:03.344679 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 03:08:03.365999 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 03:08:04.517643 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 03:08:04.517817 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ranger-kms
I20260926 03:08:04.517915 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:53695
2026-09-26 03:08:04,907 [I] DB FLAVOR :POSTGRES
2026-09-26 03:08:04,907 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 03:08:04,907 [I] Checking connection
2026-09-26 03:08:04,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:53695/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 03:08:05,160 [I] connection success
2026-09-26 03:08:05,160 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 03:08:05,160 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 03:08:05,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:53695/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 03:08:05,413 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 03:08:05,414 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 03:08:05,414 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 03:08:05,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:53695/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 03:08:05,678 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 03:08:06.206632 388 mini_ranger_kms.cc:310] Created kms service
I20260926 03:08:06.438740 388 mini_ranger_kms.cc:326] Created kudu user
I20260926 03:08:06.490880 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 03:08:06 dist-test-slave-hmbw krb5kdc[11282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392083, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:08:06.960409 388 mini_ranger_kms.cc:384] Added ranger policy
I20260926 03:08:06.960770 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 03:08:06.960834 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260926 03:08:06.979784 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11676
I20260926 03:08:06.979981 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:48429
Sep 26, 2026 3:08:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:08:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:08:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 3:08:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 3:08:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 3:08:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 03:08:09.112332 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.768s user 0.119s sys 0.294s
I20260926 03:08:09.112569 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 03:08:09.112759 388 mini_ranger_kms.cc:401] 127.0.97.20:48429/kms/v1/keys
Sep 26 03:08:09 dist-test-slave-hmbw krb5kdc[11282](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790392089, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 03:08:09 dist-test-slave-hmbw krb5kdc[11282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392089, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 03:08:10 dist-test-slave-hmbw krb5kdc[11282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392089, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 03:08:12 dist-test-slave-hmbw krb5kdc[11282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:08:12.822938 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44875
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44875
--ranger_config_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:48429/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:08:13.172801 11726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:08:13.173103 11726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:08:13.173210 11726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:08:13.180630 11726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:08:13.180753 11726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:08:13.180804 11726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:08:13.180878 11726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:08:13.180930 11726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:08:13.191272 11726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:48429/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44875
--ranger_config_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:08:13.192621 11726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:08:13.193903 11726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:08:13.208317 11731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:08:13.208325 11732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:08:13.209233 11734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:08:13.209566 11726 server_base.cc:1034] running on GCE node
I20260926 03:08:13.210402 11726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:08:13.212080 11726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:08:13.213303 11726 hybrid_clock.cc:648] HybridClock initialized: now 1790392093213248 us; error 57 us; skew 500 ppm
I20260926 03:08:13.213656 11726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:08:13 dist-test-slave-hmbw krb5kdc[11282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:08:13.216919 11726 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:08:13.219117 11726 webserver.cc:460] Webserver started at http://127.0.97.62:46825/ using document root <none> and password file <none>
I20260926 03:08:13.219836 11726 fs_manager.cc:359] Metadata directory not provided
I20260926 03:08:13.219946 11726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:08:13.220376 11726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:08:13 dist-test-slave-hmbw krb5kdc[11282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:08:13.288439 11726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "53dadc1e4ced4d6d87f1b23f46621989"
format_stamp: "Formatted at 2026-09-26 03:08:13 on dist-test-slave-hmbw"
server_key: "ff9059c9900720217b70b07e75066240"
server_key_iv: "3b47aaddc9d6dd659619b5d8e5d00f1b"
server_key_version: "kuduclusterkey@0"
I20260926 03:08:13.289289 11726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "53dadc1e4ced4d6d87f1b23f46621989"
format_stamp: "Formatted at 2026-09-26 03:08:13 on dist-test-slave-hmbw"
server_key: "ff9059c9900720217b70b07e75066240"
server_key_iv: "3b47aaddc9d6dd659619b5d8e5d00f1b"
server_key_version: "kuduclusterkey@0"
I20260926 03:08:13.294682 11726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 03:08:13.315223 11741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:08:13.316705 11726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:08:13.316877 11726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "53dadc1e4ced4d6d87f1b23f46621989"
format_stamp: "Formatted at 2026-09-26 03:08:13 on dist-test-slave-hmbw"
server_key: "ff9059c9900720217b70b07e75066240"
server_key_iv: "3b47aaddc9d6dd659619b5d8e5d00f1b"
server_key_version: "kuduclusterkey@0"
I20260926 03:08:13.317096 11726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:08:13.334331 11726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:08:13.338465 11726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:08:13.338798 11726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:08:13.359139 11726 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44875
I20260926 03:08:13.359155 11792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44875 every 8 connection(s)
I20260926 03:08:13.360522 11726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:08:13.364441 11793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:08:13.370956 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 11726
I20260926 03:08:13.371194 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:08:13.371940 388 external_mini_cluster.cc:1428] Setting key d5ba73e3ba2d0a0b515a9a545f2c486a
I20260926 03:08:13.377172 11793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53dadc1e4ced4d6d87f1b23f46621989: Bootstrap starting.
I20260926 03:08:13.381938 11793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53dadc1e4ced4d6d87f1b23f46621989: Neither blocks nor log segments found. Creating new log.
I20260926 03:08:13.383841 11793 log.cc:824] T 00000000000000000000000000000000 P 53dadc1e4ced4d6d87f1b23f46621989: Log is configured to *not* fsync() on all Append() calls
I20260926 03:08:13.386765 11793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53dadc1e4ced4d6d87f1b23f46621989: No bootstrap required, opened a new log
I20260926 03:08:13.396732 11793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53dadc1e4ced4d6d87f1b23f46621989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dadc1e4ced4d6d87f1b23f46621989" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44875 } }
I20260926 03:08:13.397140 11793 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53dadc1e4ced4d6d87f1b23f46621989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:08:13.397253 11793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53dadc1e4ced4d6d87f1b23f46621989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53dadc1e4ced4d6d87f1b23f46621989, State: Initialized, Role: FOLLOWER
Sep 26 03:08:13 dist-test-slave-hmbw krb5kdc[11282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:08:13.398120 11793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53dadc1e4ced4d6d87f1b23f46621989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dadc1e4ced4d6d87f1b23f46621989" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44875 } }
I20260926 03:08:13.398419 11793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53dadc1e4ced4d6d87f1b23f46621989 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:08:13.398641 11793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53dadc1e4ced4d6d87f1b23f46621989 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:08:13.398828 11793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53dadc1e4ced4d6d87f1b23f46621989 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:08:13.402176 11793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53dadc1e4ced4d6d87f1b23f46621989 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dadc1e4ced4d6d87f1b23f46621989" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44875 } }
I20260926 03:08:13.402848 11793 leader_election.cc:302] T 00000000000000000000000000000000 P 53dadc1e4ced4d6d87f1b23f46621989 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53dadc1e4ced4d6d87f1b23f46621989; no voters:
I20260926 03:08:13.404093 11793 leader_election.cc:288] T 00000000000000000000000000000000 P 53dadc1e4ced4d6d87f1b23f46621989 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:08:13.404206 11798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53dadc1e4ced4d6d87f1b23f46621989 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:08:13.409022 11796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:08:13.374180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56688 (local address 127.0.97.62:44875)
0926 03:08:13.382659 (+ 8479us) server_negotiation.cc:206] Beginning negotiation
0926 03:08:13.382678 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 03:08:13.383100 (+ 422us) server_negotiation.cc:407] Connection header received
0926 03:08:13.385691 (+ 2591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:08:13.385722 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:08:13.386232 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:08:13.386698 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:08:13.388301 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:08:13.390327 (+ 2026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:08:13.392656 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:08:13.393830 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:08:13.394050 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:08:13.399018 (+ 4968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:08:13.399030 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:08:13.399054 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:08:13.399116 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:08:13.402913 (+ 3797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:08:13.403912 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:08:13.403922 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:08:13.403934 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:08:13.404140 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:08:13.404635 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:08:13.404639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:08:13.404642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:08:13.405369 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:08:13.406829 (+ 1460us) server_negotiation.cc:1035] Waiting for connection context
0926 03:08:13.407108 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 03:08:13.407595 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8190,"thread_start_us":7969,"threads_started":1}
I20260926 03:08:13.411300 11798 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53dadc1e4ced4d6d87f1b23f46621989 [term 1 LEADER]: Becoming Leader. State: Replica: 53dadc1e4ced4d6d87f1b23f46621989, State: Running, Role: LEADER
I20260926 03:08:13.411963 11798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53dadc1e4ced4d6d87f1b23f46621989 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dadc1e4ced4d6d87f1b23f46621989" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44875 } }
I20260926 03:08:13.412768 11793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53dadc1e4ced4d6d87f1b23f46621989 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:08:13.415057 11800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53dadc1e4ced4d6d87f1b23f46621989 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53dadc1e4ced4d6d87f1b23f46621989. Latest consensus state: current_term: 1 leader_uuid: "53dadc1e4ced4d6d87f1b23f46621989" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dadc1e4ced4d6d87f1b23f46621989" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44875 } } }
I20260926 03:08:13.415488 11800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53dadc1e4ced4d6d87f1b23f46621989 [sys.catalog]: This master's current role is: LEADER
I20260926 03:08:13.415915 11799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53dadc1e4ced4d6d87f1b23f46621989 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53dadc1e4ced4d6d87f1b23f46621989" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53dadc1e4ced4d6d87f1b23f46621989" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44875 } } }
I20260926 03:08:13.416208 11799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53dadc1e4ced4d6d87f1b23f46621989 [sys.catalog]: This master's current role is: LEADER
I20260926 03:08:13.420200 11803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:08:13.428455 11803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:08:13.451154 11803 catalog_manager.cc:1347] Generated new cluster ID: 955b29700821476e8d6448cb3d92ce1e
I20260926 03:08:13.451292 11803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:08:13.471632 11803 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:08:13.473697 11803 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:08:13.480553 11793 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 03:08:13.492796 11803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53dadc1e4ced4d6d87f1b23f46621989: Generated new TSK 0
I20260926 03:08:13.494215 11803 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:08:14.266532 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:48429/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44875
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:08:14 dist-test-slave-hmbw krb5kdc[11282](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790392094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 03:08:14.655110 11838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:08:14.655414 11838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:08:14.655555 11838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:08:14.663558 11838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:08:14.663695 11838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:08:14.663875 11838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:08:14.676404 11838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:48429/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:08:14.677729 11838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:08:14.679045 11838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:08:14.693275 11848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:08:14.693137 11849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:08:14.693877 11851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:08:14.694725 11838 server_base.cc:1034] running on GCE node
I20260926 03:08:14.695394 11838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:08:14.696452 11838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:08:14.697700 11838 hybrid_clock.cc:648] HybridClock initialized: now 1790392094697632 us; error 60 us; skew 500 ppm
I20260926 03:08:14.698032 11838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:08:14 dist-test-slave-hmbw krb5kdc[11282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:08:14.709797 11838 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:08:14.712419 11838 webserver.cc:460] Webserver started at http://127.0.97.1:40171/ using document root <none> and password file <none>
I20260926 03:08:14.713434 11838 fs_manager.cc:359] Metadata directory not provided
I20260926 03:08:14.713660 11838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:08:14.714116 11838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:08:14 dist-test-slave-hmbw krb5kdc[11282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:08:14.740047 11838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-0/data/instance:
uuid: "b28925e9aa644ea9ba92a451c84d1b6d"
format_stamp: "Formatted at 2026-09-26 03:08:14 on dist-test-slave-hmbw"
server_key: "f827fe6d95f0631b2d0151e778d813af"
server_key_iv: "04ccc8427f184fb646797ebb9279e8a3"
server_key_version: "kuduclusterkey@0"
I20260926 03:08:14.743798 11838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b28925e9aa644ea9ba92a451c84d1b6d"
format_stamp: "Formatted at 2026-09-26 03:08:14 on dist-test-slave-hmbw"
server_key: "f827fe6d95f0631b2d0151e778d813af"
server_key_iv: "04ccc8427f184fb646797ebb9279e8a3"
server_key_version: "kuduclusterkey@0"
I20260926 03:08:14.751011 11838 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260926 03:08:14.783291 11858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:08:14.785971 11838 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.005s sys 0.007s
I20260926 03:08:14.786298 11838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "b28925e9aa644ea9ba92a451c84d1b6d"
format_stamp: "Formatted at 2026-09-26 03:08:14 on dist-test-slave-hmbw"
server_key: "f827fe6d95f0631b2d0151e778d813af"
server_key_iv: "04ccc8427f184fb646797ebb9279e8a3"
server_key_version: "kuduclusterkey@0"
I20260926 03:08:14.786700 11838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:08:14.835063 11838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:08:14.839267 11838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:08:14.839586 11838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:08:14.840664 11838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:08:14.842129 11838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:08:14.842207 11838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:08:14.842351 11838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:08:14.842404 11838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:08:14.902976 11838 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37341
I20260926 03:08:14.903023 11970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37341 every 8 connection(s)
I20260926 03:08:14.904804 11838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:08:14.911510 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 11838
I20260926 03:08:14.911722 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:08:14.912443 388 external_mini_cluster.cc:1428] Setting key d20dd447bfda4931072b7bcd52f23985
Sep 26 03:08:14 dist-test-slave-hmbw krb5kdc[11282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:08:14.951292 11974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:08:14.909871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44875 (local address 127.0.97.1:55541)
0926 03:08:14.926480 (+ 16609us) negotiation.cc:107] Waiting for socket to connect
0926 03:08:14.926568 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0926 03:08:14.929712 (+ 3144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:08:14.930839 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:08:14.930853 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:08:14.931551 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:08:14.932441 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:08:14.932474 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:08:14.934205 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:08:14.934211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:08:14.936294 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:08:14.936312 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:08:14.937679 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:08:14.937686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:08:14.937992 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:08:14.938850 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:08:14.938888 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:08:14.941738 (+ 2850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:08:14.945994 (+ 4256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:08:14.946005 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:08:14.946027 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:08:14.946770 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:08:14.947297 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:08:14.947303 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:08:14.947306 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:08:14.947641 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:08:14.948422 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:08:14.948445 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:08:14.948929 (+ 484us) client_negotiation.cc:769] Sending connection context
0926 03:08:14.949255 (+ 326us) client_negotiation.cc:240] Negotiation successful
0926 03:08:14.949835 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16269,"thread_start_us":15988,"threads_started":1}
I20260926 03:08:14.952606 11973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:08:14.909191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55541 (local address 127.0.97.62:44875)
0926 03:08:14.929824 (+ 20633us) server_negotiation.cc:206] Beginning negotiation
0926 03:08:14.929830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:08:14.929875 (+ 45us) server_negotiation.cc:407] Connection header received
0926 03:08:14.930226 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:08:14.930232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:08:14.930342 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:08:14.932659 (+ 2317us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:08:14.932838 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:08:14.934008 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:08:14.936525 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:08:14.937212 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:08:14.937343 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:08:14.942082 (+ 4739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:08:14.942090 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:08:14.942096 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:08:14.942142 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:08:14.945755 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:08:14.946961 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:08:14.946968 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:08:14.946971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:08:14.947143 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:08:14.947795 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:08:14.947801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:08:14.947804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:08:14.948190 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:08:14.948340 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 03:08:14.951882 (+ 3542us) server_negotiation.cc:299] Negotiation successful
0926 03:08:14.952146 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20489,"thread_start_us":20673,"threads_started":1}
I20260926 03:08:14.954953 11971 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44875
I20260926 03:08:14.955273 11971 heartbeater.cc:464] Registering TS with master...
I20260926 03:08:14.956156 11971 heartbeater.cc:511] Master 127.0.97.62:44875 requested a full tablet report, sending...
I20260926 03:08:14.958978 11758 ts_manager.cc:194] Registered new tserver with Master: b28925e9aa644ea9ba92a451c84d1b6d (127.0.97.1:37341)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:08:14.961813 11758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:08:14.995044 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:48429/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44875
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:08:15 dist-test-slave-hmbw krb5kdc[11282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 03:08:15 dist-test-slave-hmbw krb5kdc[11282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260926 03:08:15.491019 11977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:08:15.491779 11977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:08:15.492019 11977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:08:15.506341 11977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:08:15.506660 11977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:08:15.506915 11977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:08:15.524008 11977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:48429/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:08:15.526230 11977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:08:15.530280 11977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:08:15.581804 11985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:08:15.582647 11987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:08:15.587045 11984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:08:15.588608 11977 server_base.cc:1034] running on GCE node
I20260926 03:08:15.590051 11977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:08:15.591472 11977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:08:15.593021 11977 hybrid_clock.cc:648] HybridClock initialized: now 1790392095592909 us; error 93 us; skew 500 ppm
I20260926 03:08:15.593652 11977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:08:15 dist-test-slave-hmbw krb5kdc[11282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:08:15.598438 11977 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:08:15.600414 11977 webserver.cc:460] Webserver started at http://127.0.97.2:39671/ using document root <none> and password file <none>
I20260926 03:08:15.601720 11977 fs_manager.cc:359] Metadata directory not provided
I20260926 03:08:15.601980 11977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:08:15.602732 11977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:08:15 dist-test-slave-hmbw krb5kdc[11282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:08:15.628034 11977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-1/data/instance:
uuid: "e917cc007e3646259a70e253ffa96fb3"
format_stamp: "Formatted at 2026-09-26 03:08:15 on dist-test-slave-hmbw"
server_key: "1a5c287ae641eebb6c2f85ba5e703afa"
server_key_iv: "1af74d06fcab49240952026fbb46b3c2"
server_key_version: "kuduclusterkey@0"
I20260926 03:08:15.629528 11977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e917cc007e3646259a70e253ffa96fb3"
format_stamp: "Formatted at 2026-09-26 03:08:15 on dist-test-slave-hmbw"
server_key: "1a5c287ae641eebb6c2f85ba5e703afa"
server_key_iv: "1af74d06fcab49240952026fbb46b3c2"
server_key_version: "kuduclusterkey@0"
I20260926 03:08:15.640923 11977 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.002s sys 0.007s
I20260926 03:08:15.678798 11994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:08:15.681182 11977 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.007s sys 0.006s
I20260926 03:08:15.681536 11977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "e917cc007e3646259a70e253ffa96fb3"
format_stamp: "Formatted at 2026-09-26 03:08:15 on dist-test-slave-hmbw"
server_key: "1a5c287ae641eebb6c2f85ba5e703afa"
server_key_iv: "1af74d06fcab49240952026fbb46b3c2"
server_key_version: "kuduclusterkey@0"
I20260926 03:08:15.681932 11977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:08:15.736517 11977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:08:15.753290 11977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:08:15.753726 11977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:08:15.771205 11977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:08:15.801317 11977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:08:15.802079 11977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 03:08:15.802335 11977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:08:15.802453 11977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:08:15.869479 11977 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44693
I20260926 03:08:15.869509 12113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44693 every 8 connection(s)
I20260926 03:08:15.871373 11977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:08:15.873713 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 11977
I20260926 03:08:15.873885 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:08:15.874408 388 external_mini_cluster.cc:1428] Setting key 30760250cc6bc4914605af90745a10d0
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Sep 26 03:08:15 dist-test-slave-hmbw krb5kdc[11282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:08:15.931581 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:08:15.885048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54503 (local address 127.0.97.62:44875)
0926 03:08:15.901435 (+ 16387us) server_negotiation.cc:206] Beginning negotiation
0926 03:08:15.901442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:08:15.902810 (+ 1368us) server_negotiation.cc:407] Connection header received
0926 03:08:15.912800 (+ 9990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:08:15.912806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:08:15.912904 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:08:15.913001 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:08:15.915281 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:08:15.916344 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:08:15.918812 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:08:15.919403 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:08:15.919514 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:08:15.924088 (+ 4574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:08:15.924096 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:08:15.924101 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:08:15.924145 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:08:15.927500 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:08:15.928593 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:08:15.928601 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:08:15.928605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:08:15.928762 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:08:15.929298 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:08:15.929303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:08:15.929306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:08:15.929681 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:08:15.929818 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 03:08:15.930858 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0926 03:08:15.931175 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16244,"thread_start_us":16110,"threads_started":1}
I20260926 03:08:15.932569 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:08:15.884417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44875 (local address 127.0.97.2:54503)
0926 03:08:15.902218 (+ 17801us) negotiation.cc:107] Waiting for socket to connect
0926 03:08:15.902283 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 03:08:15.912260 (+ 9977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:08:15.913522 (+ 1262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:08:15.913535 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:08:15.914163 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:08:15.915060 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:08:15.915086 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:08:15.916608 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:08:15.916613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:08:15.918610 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:08:15.918629 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:08:15.919868 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:08:15.919875 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:08:15.920155 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:08:15.921016 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:08:15.921048 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:08:15.923781 (+ 2733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:08:15.927757 (+ 3976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:08:15.927767 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:08:15.927789 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:08:15.928416 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:08:15.928887 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:08:15.928892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:08:15.928896 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:08:15.929178 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:08:15.929906 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:08:15.929914 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:08:15.930346 (+ 432us) client_negotiation.cc:769] Sending connection context
0926 03:08:15.930702 (+ 356us) client_negotiation.cc:240] Negotiation successful
0926 03:08:15.931241 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17447,"thread_start_us":17181,"threads_started":1}
I20260926 03:08:15.934033 12114 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44875
I20260926 03:08:15.934358 12114 heartbeater.cc:464] Registering TS with master...
I20260926 03:08:15.935261 12114 heartbeater.cc:511] Master 127.0.97.62:44875 requested a full tablet report, sending...
I20260926 03:08:15.936968 11758 ts_manager.cc:194] Registered new tserver with Master: e917cc007e3646259a70e253ffa96fb3 (127.0.97.2:44693)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:08:15.938143 11758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54503
I20260926 03:08:15.939494 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:48429/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44875
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:08:15.966164 11971 heartbeater.cc:503] Master 127.0.97.62:44875 was elected leader, sending a full tablet report...
W20260926 03:08:16.361382 12120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:08:16.361673 12120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:08:16.361789 12120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:08:16.369688 12120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:08:16.369835 12120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:08:16.369992 12120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:08:16.380064 12120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:48429/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:08:16.381381 12120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:08:16.382741 12120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:08:16.396613 12125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:08:16.396555 12128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:08:16.396595 12126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:08:16.397511 12120 server_base.cc:1034] running on GCE node
I20260926 03:08:16.398171 12120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:08:16.399281 12120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:08:16.400560 12120 hybrid_clock.cc:648] HybridClock initialized: now 1790392096400467 us; error 86 us; skew 500 ppm
I20260926 03:08:16.400950 12120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:08:16 dist-test-slave-hmbw krb5kdc[11282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:08:16.404330 12120 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:08:16.405800 12120 webserver.cc:460] Webserver started at http://127.0.97.3:34893/ using document root <none> and password file <none>
I20260926 03:08:16.406441 12120 fs_manager.cc:359] Metadata directory not provided
I20260926 03:08:16.406594 12120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:08:16.406906 12120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:08:16 dist-test-slave-hmbw krb5kdc[11282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:08:16.425830 12120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-2/data/instance:
uuid: "1fae15d032d14387aece7c45cde5b19c"
format_stamp: "Formatted at 2026-09-26 03:08:16 on dist-test-slave-hmbw"
server_key: "18055379104a7c535f9a30a3b6bbf27c"
server_key_iv: "fd309ba7b880f8bca55b5c78bf7fbf59"
server_key_version: "kuduclusterkey@0"
I20260926 03:08:16.427608 12120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1fae15d032d14387aece7c45cde5b19c"
format_stamp: "Formatted at 2026-09-26 03:08:16 on dist-test-slave-hmbw"
server_key: "18055379104a7c535f9a30a3b6bbf27c"
server_key_iv: "fd309ba7b880f8bca55b5c78bf7fbf59"
server_key_version: "kuduclusterkey@0"
I20260926 03:08:16.434965 12120 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260926 03:08:16.453729 12135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:08:16.455703 12120 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:08:16.455920 12120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "1fae15d032d14387aece7c45cde5b19c"
format_stamp: "Formatted at 2026-09-26 03:08:16 on dist-test-slave-hmbw"
server_key: "18055379104a7c535f9a30a3b6bbf27c"
server_key_iv: "fd309ba7b880f8bca55b5c78bf7fbf59"
server_key_version: "kuduclusterkey@0"
I20260926 03:08:16.456210 12120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:08:16.484608 12120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:08:16.489143 12120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:08:16.489403 12120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:08:16.490339 12120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:08:16.491868 12120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:08:16.491950 12120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:08:16.492056 12120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:08:16.492096 12120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:08:16.526111 12120 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33341
I20260926 03:08:16.526139 12247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33341 every 8 connection(s)
I20260926 03:08:16.527500 12120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:08:16.529688 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 12120
I20260926 03:08:16.529842 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:08:16.530364 388 external_mini_cluster.cc:1428] Setting key 322f79533a60567975b01a899c91d856
Sep 26 03:08:16 dist-test-slave-hmbw krb5kdc[11282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:08:16.547655 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:08:16.530088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45705 (local address 127.0.97.62:44875)
0926 03:08:16.530638 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0926 03:08:16.530644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:08:16.531681 (+ 1037us) server_negotiation.cc:407] Connection header received
0926 03:08:16.533766 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:08:16.533772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:08:16.533887 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:08:16.533999 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:08:16.535541 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.536274 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.537922 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.538335 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.538471 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:08:16.541789 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:08:16.541795 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:08:16.541798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:08:16.541833 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:08:16.544533 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:08:16.545346 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:08:16.545352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:08:16.545354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:08:16.545515 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:08:16.545986 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:08:16.545991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:08:16.545995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:08:16.546349 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:08:16.546480 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 03:08:16.547220 (+ 740us) server_negotiation.cc:299] Negotiation successful
0926 03:08:16.547387 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":214,"threads_started":1}
I20260926 03:08:16.548570 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:08:16.530614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44875 (local address 127.0.97.3:45705)
0926 03:08:16.531513 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0926 03:08:16.531550 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 03:08:16.533493 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:08:16.534279 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:08:16.534288 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:08:16.534778 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:08:16.535366 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:08:16.535383 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.536417 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.536420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:08:16.537717 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:08:16.537730 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.538588 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.538596 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:08:16.538802 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:08:16.539408 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:08:16.539430 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:08:16.541547 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:08:16.544702 (+ 3155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:08:16.544711 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:08:16.544738 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:08:16.545211 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:08:16.545624 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:08:16.545627 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:08:16.545630 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:08:16.545874 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:08:16.546551 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:08:16.546559 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:08:16.546897 (+ 338us) client_negotiation.cc:769] Sending connection context
0926 03:08:16.547129 (+ 232us) client_negotiation.cc:240] Negotiation successful
0926 03:08:16.547535 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":390,"threads_started":1}
I20260926 03:08:16.550009 12248 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44875
I20260926 03:08:16.550302 12248 heartbeater.cc:464] Registering TS with master...
I20260926 03:08:16.551017 12248 heartbeater.cc:511] Master 127.0.97.62:44875 requested a full tablet report, sending...
I20260926 03:08:16.552309 11758 ts_manager.cc:194] Registered new tserver with Master: 1fae15d032d14387aece7c45cde5b19c (127.0.97.3:33341)
I20260926 03:08:16.553117 11758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45705
I20260926 03:08:16.559427 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:08:16.575728 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:08:16.563822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54148 (local address 127.0.97.62:44875)
0926 03:08:16.563998 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 03:08:16.564003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:08:16.564484 (+ 481us) server_negotiation.cc:407] Connection header received
0926 03:08:16.564667 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:08:16.564671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:08:16.564748 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:08:16.564832 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:08:16.566128 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.566944 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.568267 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.568709 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.568810 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:08:16.571029 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:08:16.571034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:08:16.571038 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:08:16.571069 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:08:16.573385 (+ 2316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:08:16.574052 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:08:16.574056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:08:16.574059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:08:16.574191 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:08:16.574546 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:08:16.574551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:08:16.574554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:08:16.574824 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:08:16.575012 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 03:08:16.575292 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 03:08:16.575505 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 03:08:16.579734 11758 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:08:16.587363 11758 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:08:16.622238 12264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:08:16.600186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42904 (local address 127.0.97.1:37341)
0926 03:08:16.614289 (+ 14103us) server_negotiation.cc:206] Beginning negotiation
0926 03:08:16.614298 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:08:16.614328 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:08:16.614424 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:08:16.614433 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:08:16.614711 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:08:16.614894 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:08:16.616074 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.617261 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.619569 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.621064 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.621187 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:08:16.621435 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 03:08:16.621714 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 03:08:16.621886 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13943,"thread_start_us":13802,"threads_started":1}
I20260926 03:08:16.622320 12262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:08:16.600070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37341 (local address 127.0.0.1:42904)
0926 03:08:16.613619 (+ 13549us) negotiation.cc:107] Waiting for socket to connect
0926 03:08:16.613646 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 03:08:16.613768 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:08:16.615500 (+ 1732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:08:16.615505 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:08:16.615528 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:08:16.615863 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:08:16.615875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.617393 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.617396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:08:16.619390 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:08:16.619404 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.621251 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.621257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:08:16.621453 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:08:16.621514 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 03:08:16.621645 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 03:08:16.621886 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13411,"mutex_wait_us":1,"thread_start_us":13203,"threads_started":1}
I20260926 03:08:16.622442 12261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:08:16.599723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33341 (local address 127.0.0.1:47134)
0926 03:08:16.601186 (+ 1463us) negotiation.cc:107] Waiting for socket to connect
0926 03:08:16.601226 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 03:08:16.601417 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:08:16.614295 (+ 12878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:08:16.614303 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:08:16.614337 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:08:16.614835 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:08:16.614854 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.616577 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.616583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:08:16.619087 (+ 2504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:08:16.619100 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.621584 (+ 2484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.621591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:08:16.621751 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:08:16.621788 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 03:08:16.621856 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 03:08:16.621983 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":77,"thread_start_us":591,"threads_started":1}
I20260926 03:08:16.623450 12263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:08:16.600083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47134 (local address 127.0.97.3:33341)
0926 03:08:16.612352 (+ 12269us) server_negotiation.cc:206] Beginning negotiation
0926 03:08:16.612363 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:08:16.612408 (+ 45us) server_negotiation.cc:407] Connection header received
0926 03:08:16.612546 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:08:16.612556 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:08:16.612878 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:08:16.614254 (+ 1376us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:08:16.615082 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.616445 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.619281 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.621343 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.622678 (+ 1335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:08:16.622903 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 03:08:16.622966 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 03:08:16.623110 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12049,"thread_start_us":13910,"threads_started":1}
I20260926 03:08:16.623656 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:08:16.602157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39678 (local address 127.0.97.2:44693)
0926 03:08:16.613962 (+ 11805us) server_negotiation.cc:206] Beginning negotiation
0926 03:08:16.613970 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:08:16.615190 (+ 1220us) server_negotiation.cc:407] Connection header received
0926 03:08:16.615490 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:08:16.615498 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:08:16.615717 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:08:16.615863 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:08:16.616621 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.618067 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.620276 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.621943 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.622052 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:08:16.622233 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 03:08:16.622800 (+ 567us) server_negotiation.cc:299] Negotiation successful
0926 03:08:16.623373 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11658,"thread_start_us":11552,"threads_started":1}
I20260926 03:08:16.624358 12265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:08:16.601928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44693 (local address 127.0.0.1:39678)
0926 03:08:16.615086 (+ 13158us) negotiation.cc:107] Waiting for socket to connect
0926 03:08:16.615104 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:08:16.615293 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:08:16.616045 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:08:16.616049 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:08:16.616072 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:08:16.616432 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:08:16.616444 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.618211 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.618217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:08:16.620133 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:08:16.620147 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.622420 (+ 2273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.622430 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:08:16.622613 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:08:16.622643 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 03:08:16.623875 (+ 1232us) client_negotiation.cc:240] Negotiation successful
0926 03:08:16.624034 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13024,"thread_start_us":12888,"threads_started":1}
I20260926 03:08:16.626246 11906 tablet_service.cc:1467] Processing CreateTablet for tablet 4fc9ff7607d14549a5d370b9f82a0ada (DEFAULT_TABLE table=test-table [id=5d1981ba54cb486080d24f47dca662c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:08:16.627640 11906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fc9ff7607d14549a5d370b9f82a0ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:08:16.627285 12049 tablet_service.cc:1467] Processing CreateTablet for tablet 4fc9ff7607d14549a5d370b9f82a0ada (DEFAULT_TABLE table=test-table [id=5d1981ba54cb486080d24f47dca662c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:08:16.628685 12049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fc9ff7607d14549a5d370b9f82a0ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:08:16.629153 12183 tablet_service.cc:1467] Processing CreateTablet for tablet 4fc9ff7607d14549a5d370b9f82a0ada (DEFAULT_TABLE table=test-table [id=5d1981ba54cb486080d24f47dca662c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:08:16.630529 12183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fc9ff7607d14549a5d370b9f82a0ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:08:16.639452 12267 tablet_bootstrap.cc:492] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d: Bootstrap starting.
I20260926 03:08:16.643225 12267 tablet_bootstrap.cc:654] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d: Neither blocks nor log segments found. Creating new log.
I20260926 03:08:16.643733 12268 tablet_bootstrap.cc:492] T 4fc9ff7607d14549a5d370b9f82a0ada P e917cc007e3646259a70e253ffa96fb3: Bootstrap starting.
I20260926 03:08:16.644749 12267 log.cc:824] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d: Log is configured to *not* fsync() on all Append() calls
I20260926 03:08:16.645871 12269 tablet_bootstrap.cc:492] T 4fc9ff7607d14549a5d370b9f82a0ada P 1fae15d032d14387aece7c45cde5b19c: Bootstrap starting.
I20260926 03:08:16.647707 12268 tablet_bootstrap.cc:654] T 4fc9ff7607d14549a5d370b9f82a0ada P e917cc007e3646259a70e253ffa96fb3: Neither blocks nor log segments found. Creating new log.
I20260926 03:08:16.649094 12268 log.cc:824] T 4fc9ff7607d14549a5d370b9f82a0ada P e917cc007e3646259a70e253ffa96fb3: Log is configured to *not* fsync() on all Append() calls
I20260926 03:08:16.649221 12269 tablet_bootstrap.cc:654] T 4fc9ff7607d14549a5d370b9f82a0ada P 1fae15d032d14387aece7c45cde5b19c: Neither blocks nor log segments found. Creating new log.
I20260926 03:08:16.650420 12269 log.cc:824] T 4fc9ff7607d14549a5d370b9f82a0ada P 1fae15d032d14387aece7c45cde5b19c: Log is configured to *not* fsync() on all Append() calls
I20260926 03:08:16.652098 12268 tablet_bootstrap.cc:492] T 4fc9ff7607d14549a5d370b9f82a0ada P e917cc007e3646259a70e253ffa96fb3: No bootstrap required, opened a new log
I20260926 03:08:16.652101 12267 tablet_bootstrap.cc:492] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d: No bootstrap required, opened a new log
I20260926 03:08:16.652401 12268 ts_tablet_manager.cc:1397] T 4fc9ff7607d14549a5d370b9f82a0ada P e917cc007e3646259a70e253ffa96fb3: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 03:08:16.652401 12267 ts_tablet_manager.cc:1397] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20260926 03:08:16.652768 12269 tablet_bootstrap.cc:492] T 4fc9ff7607d14549a5d370b9f82a0ada P 1fae15d032d14387aece7c45cde5b19c: No bootstrap required, opened a new log
I20260926 03:08:16.653008 12269 ts_tablet_manager.cc:1397] T 4fc9ff7607d14549a5d370b9f82a0ada P 1fae15d032d14387aece7c45cde5b19c: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260926 03:08:16.660703 12269 raft_consensus.cc:348] T 4fc9ff7607d14549a5d370b9f82a0ada P 1fae15d032d14387aece7c45cde5b19c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fae15d032d14387aece7c45cde5b19c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33341 } } peers { permanent_uuid: "b28925e9aa644ea9ba92a451c84d1b6d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37341 } } peers { permanent_uuid: "e917cc007e3646259a70e253ffa96fb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44693 } }
I20260926 03:08:16.661167 12269 raft_consensus.cc:374] T 4fc9ff7607d14549a5d370b9f82a0ada P 1fae15d032d14387aece7c45cde5b19c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:08:16.661296 12269 raft_consensus.cc:729] T 4fc9ff7607d14549a5d370b9f82a0ada P 1fae15d032d14387aece7c45cde5b19c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fae15d032d14387aece7c45cde5b19c, State: Initialized, Role: FOLLOWER
I20260926 03:08:16.661978 12269 consensus_queue.cc:260] T 4fc9ff7607d14549a5d370b9f82a0ada P 1fae15d032d14387aece7c45cde5b19c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fae15d032d14387aece7c45cde5b19c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33341 } } peers { permanent_uuid: "b28925e9aa644ea9ba92a451c84d1b6d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37341 } } peers { permanent_uuid: "e917cc007e3646259a70e253ffa96fb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44693 } }
I20260926 03:08:16.663352 12248 heartbeater.cc:503] Master 127.0.97.62:44875 was elected leader, sending a full tablet report...
I20260926 03:08:16.663026 12268 raft_consensus.cc:348] T 4fc9ff7607d14549a5d370b9f82a0ada P e917cc007e3646259a70e253ffa96fb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fae15d032d14387aece7c45cde5b19c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33341 } } peers { permanent_uuid: "b28925e9aa644ea9ba92a451c84d1b6d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37341 } } peers { permanent_uuid: "e917cc007e3646259a70e253ffa96fb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44693 } }
I20260926 03:08:16.663609 12268 raft_consensus.cc:374] T 4fc9ff7607d14549a5d370b9f82a0ada P e917cc007e3646259a70e253ffa96fb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:08:16.663419 12267 raft_consensus.cc:348] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fae15d032d14387aece7c45cde5b19c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33341 } } peers { permanent_uuid: "b28925e9aa644ea9ba92a451c84d1b6d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37341 } } peers { permanent_uuid: "e917cc007e3646259a70e253ffa96fb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44693 } }
I20260926 03:08:16.663697 12268 raft_consensus.cc:729] T 4fc9ff7607d14549a5d370b9f82a0ada P e917cc007e3646259a70e253ffa96fb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e917cc007e3646259a70e253ffa96fb3, State: Initialized, Role: FOLLOWER
I20260926 03:08:16.663810 12267 raft_consensus.cc:374] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:08:16.663887 12267 raft_consensus.cc:729] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b28925e9aa644ea9ba92a451c84d1b6d, State: Initialized, Role: FOLLOWER
I20260926 03:08:16.663966 12269 ts_tablet_manager.cc:1428] T 4fc9ff7607d14549a5d370b9f82a0ada P 1fae15d032d14387aece7c45cde5b19c: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260926 03:08:16.664424 12267 consensus_queue.cc:260] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fae15d032d14387aece7c45cde5b19c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33341 } } peers { permanent_uuid: "b28925e9aa644ea9ba92a451c84d1b6d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37341 } } peers { permanent_uuid: "e917cc007e3646259a70e253ffa96fb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44693 } }
I20260926 03:08:16.664423 12268 consensus_queue.cc:260] T 4fc9ff7607d14549a5d370b9f82a0ada P e917cc007e3646259a70e253ffa96fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fae15d032d14387aece7c45cde5b19c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33341 } } peers { permanent_uuid: "b28925e9aa644ea9ba92a451c84d1b6d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37341 } } peers { permanent_uuid: "e917cc007e3646259a70e253ffa96fb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44693 } }
I20260926 03:08:16.667912 12267 ts_tablet_manager.cc:1428] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260926 03:08:16.668447 12114 heartbeater.cc:503] Master 127.0.97.62:44875 was elected leader, sending a full tablet report...
I20260926 03:08:16.671913 12268 ts_tablet_manager.cc:1428] T 4fc9ff7607d14549a5d370b9f82a0ada P e917cc007e3646259a70e253ffa96fb3: Time spent starting tablet: real 0.019s user 0.011s sys 0.003s
I20260926 03:08:16.707100 12275 raft_consensus.cc:482] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:08:16.707401 12275 raft_consensus.cc:504] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fae15d032d14387aece7c45cde5b19c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33341 } } peers { permanent_uuid: "b28925e9aa644ea9ba92a451c84d1b6d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37341 } } peers { permanent_uuid: "e917cc007e3646259a70e253ffa96fb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44693 } }
I20260926 03:08:16.709470 12275 leader_election.cc:288] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fae15d032d14387aece7c45cde5b19c (127.0.97.3:33341), e917cc007e3646259a70e253ffa96fb3 (127.0.97.2:44693)
I20260926 03:08:16.717139 12263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:08:16.709920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36755 (local address 127.0.97.3:33341)
0926 03:08:16.710094 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 03:08:16.710099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:08:16.711285 (+ 1186us) server_negotiation.cc:407] Connection header received
0926 03:08:16.711459 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:08:16.711464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:08:16.711550 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:08:16.711680 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:08:16.712442 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.713183 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.715426 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.716222 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.716359 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:08:16.716471 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 03:08:16.716754 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 03:08:16.716880 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 03:08:16.717273 12276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:08:16.709869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33341 (local address 127.0.97.1:36755)
0926 03:08:16.711137 (+ 1268us) negotiation.cc:107] Waiting for socket to connect
0926 03:08:16.711157 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:08:16.711309 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:08:16.711869 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:08:16.711874 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:08:16.711935 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:08:16.712297 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:08:16.712308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.713314 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.713320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:08:16.715211 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:08:16.715227 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.716443 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.716449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:08:16.716604 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:08:16.716628 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 03:08:16.716711 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 03:08:16.716845 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1138,"mutex_wait_us":23,"thread_start_us":955,"threads_started":1}
I20260926 03:08:16.717597 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:08:16.711699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60325 (local address 127.0.97.2:44693)
0926 03:08:16.711883 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 03:08:16.711887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:08:16.712080 (+ 193us) server_negotiation.cc:407] Connection header received
0926 03:08:16.712331 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:08:16.712335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:08:16.712491 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:08:16.712602 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:08:16.713103 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.713820 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.715901 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.716694 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.716812 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:08:16.716904 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 03:08:16.717222 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 03:08:16.717357 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:08:16.717706 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:08:16.711324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44693 (local address 127.0.97.1:60325)
0926 03:08:16.711971 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0926 03:08:16.711994 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:08:16.712128 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:08:16.712620 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:08:16.712625 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:08:16.712649 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:08:16.712961 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:08:16.712970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.713968 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.713975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:08:16.715746 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:08:16.715758 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.716932 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.716939 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:08:16.717080 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:08:16.717103 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 03:08:16.717186 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 03:08:16.717320 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":24,"thread_start_us":317,"threads_started":1}
I20260926 03:08:16.718355 12203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc9ff7607d14549a5d370b9f82a0ada" candidate_uuid: "b28925e9aa644ea9ba92a451c84d1b6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fae15d032d14387aece7c45cde5b19c" is_pre_election: true
I20260926 03:08:16.718685 12069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc9ff7607d14549a5d370b9f82a0ada" candidate_uuid: "b28925e9aa644ea9ba92a451c84d1b6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e917cc007e3646259a70e253ffa96fb3" is_pre_election: true
I20260926 03:08:16.718966 12203 raft_consensus.cc:2454] T 4fc9ff7607d14549a5d370b9f82a0ada P 1fae15d032d14387aece7c45cde5b19c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b28925e9aa644ea9ba92a451c84d1b6d in term 0.
I20260926 03:08:16.719147 12069 raft_consensus.cc:2454] T 4fc9ff7607d14549a5d370b9f82a0ada P e917cc007e3646259a70e253ffa96fb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b28925e9aa644ea9ba92a451c84d1b6d in term 0.
I20260926 03:08:16.719743 11862 leader_election.cc:302] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fae15d032d14387aece7c45cde5b19c, b28925e9aa644ea9ba92a451c84d1b6d; no voters:
I20260926 03:08:16.720223 12275 raft_consensus.cc:2781] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:08:16.720391 12275 raft_consensus.cc:482] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:08:16.720502 12275 raft_consensus.cc:3037] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:08:16.722846 12275 raft_consensus.cc:504] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fae15d032d14387aece7c45cde5b19c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33341 } } peers { permanent_uuid: "b28925e9aa644ea9ba92a451c84d1b6d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37341 } } peers { permanent_uuid: "e917cc007e3646259a70e253ffa96fb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44693 } }
I20260926 03:08:16.723412 12275 leader_election.cc:288] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d [CANDIDATE]: Term 1 election: Requested vote from peers 1fae15d032d14387aece7c45cde5b19c (127.0.97.3:33341), e917cc007e3646259a70e253ffa96fb3 (127.0.97.2:44693)
I20260926 03:08:16.723896 12203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc9ff7607d14549a5d370b9f82a0ada" candidate_uuid: "b28925e9aa644ea9ba92a451c84d1b6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fae15d032d14387aece7c45cde5b19c"
I20260926 03:08:16.723996 12069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc9ff7607d14549a5d370b9f82a0ada" candidate_uuid: "b28925e9aa644ea9ba92a451c84d1b6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e917cc007e3646259a70e253ffa96fb3"
I20260926 03:08:16.724114 12203 raft_consensus.cc:3037] T 4fc9ff7607d14549a5d370b9f82a0ada P 1fae15d032d14387aece7c45cde5b19c [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:08:16.724190 12069 raft_consensus.cc:3037] T 4fc9ff7607d14549a5d370b9f82a0ada P e917cc007e3646259a70e253ffa96fb3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:08:16.727278 12203 raft_consensus.cc:2454] T 4fc9ff7607d14549a5d370b9f82a0ada P 1fae15d032d14387aece7c45cde5b19c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b28925e9aa644ea9ba92a451c84d1b6d in term 1.
I20260926 03:08:16.727278 12069 raft_consensus.cc:2454] T 4fc9ff7607d14549a5d370b9f82a0ada P e917cc007e3646259a70e253ffa96fb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b28925e9aa644ea9ba92a451c84d1b6d in term 1.
I20260926 03:08:16.727797 11862 leader_election.cc:302] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b28925e9aa644ea9ba92a451c84d1b6d, e917cc007e3646259a70e253ffa96fb3; no voters:
I20260926 03:08:16.728135 12275 raft_consensus.cc:2781] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:08:16.728616 12275 raft_consensus.cc:686] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d [term 1 LEADER]: Becoming Leader. State: Replica: b28925e9aa644ea9ba92a451c84d1b6d, State: Running, Role: LEADER
I20260926 03:08:16.729157 12275 consensus_queue.cc:237] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fae15d032d14387aece7c45cde5b19c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33341 } } peers { permanent_uuid: "b28925e9aa644ea9ba92a451c84d1b6d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37341 } } peers { permanent_uuid: "e917cc007e3646259a70e253ffa96fb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44693 } }
I20260926 03:08:16.733830 11758 catalog_manager.cc:5495] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d reported cstate change: term changed from 0 to 1, leader changed from <none> to b28925e9aa644ea9ba92a451c84d1b6d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "b28925e9aa644ea9ba92a451c84d1b6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fae15d032d14387aece7c45cde5b19c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b28925e9aa644ea9ba92a451c84d1b6d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e917cc007e3646259a70e253ffa96fb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44693 } health_report { overall_health: UNKNOWN } } }
I20260926 03:08:16.752957 12264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:08:16.746865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42908 (local address 127.0.97.1:37341)
0926 03:08:16.747088 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 03:08:16.747093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:08:16.747127 (+ 34us) server_negotiation.cc:407] Connection header received
0926 03:08:16.747322 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:08:16.747326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:08:16.747410 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:08:16.747540 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:08:16.747927 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.748687 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.750571 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.751026 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.751124 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:08:16.751627 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:08:16.752246 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:08:16.752390 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 03:08:16.752579 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 03:08:16.752656 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:08:16.761270 12069 raft_consensus.cc:1261] T 4fc9ff7607d14549a5d370b9f82a0ada P e917cc007e3646259a70e253ffa96fb3 [term 1 FOLLOWER]: Refusing update from remote peer b28925e9aa644ea9ba92a451c84d1b6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:08:16.761298 12203 raft_consensus.cc:1261] T 4fc9ff7607d14549a5d370b9f82a0ada P 1fae15d032d14387aece7c45cde5b19c [term 1 FOLLOWER]: Refusing update from remote peer b28925e9aa644ea9ba92a451c84d1b6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:08:16.762354 12278 consensus_queue.cc:1035] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fae15d032d14387aece7c45cde5b19c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:08:16.762845 12275 consensus_queue.cc:1035] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e917cc007e3646259a70e253ffa96fb3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:08:16.775105 12283 mvcc.cc:205] Tried to move back new op lower bound from 7333446028323381248 to 7333446028206645248. Current Snapshot: MvccSnapshot[applied={T|T < 7333446028323381248}]
I20260926 03:08:16.777436 12282 mvcc.cc:205] Tried to move back new op lower bound from 7333446028323381248 to 7333446028206645248. Current Snapshot: MvccSnapshot[applied={T|T < 7333446028323381248}]
I20260926 03:08:16.776150 12281 mvcc.cc:205] Tried to move back new op lower bound from 7333446028323381248 to 7333446028206645248. Current Snapshot: MvccSnapshot[applied={T|T < 7333446028323381248}]
I20260926 03:08:16.792681 11757 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:08:16.793045 11757 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:08:16.797319 11757 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 53dadc1e4ced4d6d87f1b23f46621989: Sending DeleteTablet for 3 replicas of tablet 4fc9ff7607d14549a5d370b9f82a0ada
I20260926 03:08:16.798601 11906 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fc9ff7607d14549a5d370b9f82a0ada with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:08:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42904
I20260926 03:08:16.798592 12183 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fc9ff7607d14549a5d370b9f82a0ada with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:08:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47134
I20260926 03:08:16.799610 12049 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fc9ff7607d14549a5d370b9f82a0ada with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:08:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39678
I20260926 03:08:16.799908 12291 tablet_replica.cc:330] T 4fc9ff7607d14549a5d370b9f82a0ada P 1fae15d032d14387aece7c45cde5b19c: stopping tablet replica
I20260926 03:08:16.800025 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 11838
I20260926 03:08:16.800350 12291 raft_consensus.cc:2229] T 4fc9ff7607d14549a5d370b9f82a0ada P 1fae15d032d14387aece7c45cde5b19c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:08:16.801460 12291 raft_consensus.cc:2258] T 4fc9ff7607d14549a5d370b9f82a0ada P 1fae15d032d14387aece7c45cde5b19c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:08:16.803745 12291 ts_tablet_manager.cc:1905] T 4fc9ff7607d14549a5d370b9f82a0ada P 1fae15d032d14387aece7c45cde5b19c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:08:16.810477 12292 tablet_replica.cc:330] T 4fc9ff7607d14549a5d370b9f82a0ada P e917cc007e3646259a70e253ffa96fb3: stopping tablet replica
I20260926 03:08:16.811154 12292 raft_consensus.cc:2229] T 4fc9ff7607d14549a5d370b9f82a0ada P e917cc007e3646259a70e253ffa96fb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:08:16.811784 12292 raft_consensus.cc:2258] T 4fc9ff7607d14549a5d370b9f82a0ada P e917cc007e3646259a70e253ffa96fb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:08:16.811856 12291 ts_tablet_manager.cc:1918] T 4fc9ff7607d14549a5d370b9f82a0ada P 1fae15d032d14387aece7c45cde5b19c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:08:16.812067 12291 log.cc:1196] T 4fc9ff7607d14549a5d370b9f82a0ada P 1fae15d032d14387aece7c45cde5b19c: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-2/wal/wals/4fc9ff7607d14549a5d370b9f82a0ada
I20260926 03:08:16.812631 12291 ts_tablet_manager.cc:1939] T 4fc9ff7607d14549a5d370b9f82a0ada P 1fae15d032d14387aece7c45cde5b19c: Deleting consensus metadata
I20260926 03:08:16.814201 12292 ts_tablet_manager.cc:1905] T 4fc9ff7607d14549a5d370b9f82a0ada P e917cc007e3646259a70e253ffa96fb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:08:16.814745 11745 catalog_manager.cc:4841] TS 1fae15d032d14387aece7c45cde5b19c (127.0.97.3:33341): tablet 4fc9ff7607d14549a5d370b9f82a0ada (table test-table [id=5d1981ba54cb486080d24f47dca662c1]) successfully deleted
I20260926 03:08:16.816386 12290 tablet_replica.cc:330] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d: stopping tablet replica
I20260926 03:08:16.816932 12290 raft_consensus.cc:2229] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:08:16.817737 12290 raft_consensus.cc:2258] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:08:16.820241 12290 ts_tablet_manager.cc:1905] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:08:16.821394 12292 ts_tablet_manager.cc:1918] T 4fc9ff7607d14549a5d370b9f82a0ada P e917cc007e3646259a70e253ffa96fb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:08:16.821623 12292 log.cc:1196] T 4fc9ff7607d14549a5d370b9f82a0ada P e917cc007e3646259a70e253ffa96fb3: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-1/wal/wals/4fc9ff7607d14549a5d370b9f82a0ada
I20260926 03:08:16.822152 12292 ts_tablet_manager.cc:1939] T 4fc9ff7607d14549a5d370b9f82a0ada P e917cc007e3646259a70e253ffa96fb3: Deleting consensus metadata
I20260926 03:08:16.824028 11745 catalog_manager.cc:4841] TS e917cc007e3646259a70e253ffa96fb3 (127.0.97.2:44693): tablet 4fc9ff7607d14549a5d370b9f82a0ada (table test-table [id=5d1981ba54cb486080d24f47dca662c1]) successfully deleted
Sep 26 03:08:16 dist-test-slave-hmbw krb5kdc[11282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:08:16.827958 12290 ts_tablet_manager.cc:1918] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:08:16.828135 12290 log.cc:1196] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391950058021-388-0/minicluster-data/ts-0/wal/wals/4fc9ff7607d14549a5d370b9f82a0ada
I20260926 03:08:16.828685 12290 ts_tablet_manager.cc:1939] T 4fc9ff7607d14549a5d370b9f82a0ada P b28925e9aa644ea9ba92a451c84d1b6d: Deleting consensus metadata
I20260926 03:08:16.830588 11742 catalog_manager.cc:4841] TS b28925e9aa644ea9ba92a451c84d1b6d (127.0.97.1:37341): tablet 4fc9ff7607d14549a5d370b9f82a0ada (table test-table [id=5d1981ba54cb486080d24f47dca662c1]) successfully deleted
I20260926 03:08:16.834590 12264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:08:16.802658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42924 (local address 127.0.97.1:37341)
0926 03:08:16.802838 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 03:08:16.802844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:08:16.815926 (+ 13082us) server_negotiation.cc:407] Connection header received
0926 03:08:16.816021 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:08:16.816026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:08:16.816132 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:08:16.816259 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:08:16.818292 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.819486 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.821782 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.822563 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.822670 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:08:16.828327 (+ 5657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:08:16.828343 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:08:16.828362 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:08:16.828427 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:08:16.831834 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:08:16.832522 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:08:16.832531 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:08:16.832533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:08:16.832684 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:08:16.833059 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:08:16.833063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:08:16.833065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:08:16.833429 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:08:16.833701 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 03:08:16.833978 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 03:08:16.834203 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 03:08:16.835773 11966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:08:16.934304 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 11838
I20260926 03:08:16.948863 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 11977
Sep 26 03:08:16 dist-test-slave-hmbw krb5kdc[11282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:08:16.964202 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:08:16.949584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39688 (local address 127.0.97.2:44693)
0926 03:08:16.949761 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 03:08:16.949767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:08:16.949786 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:08:16.949992 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:08:16.949996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:08:16.950081 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:08:16.950171 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:08:16.951471 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.952183 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.953515 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:08:16.953884 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:08:16.954003 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:08:16.958169 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:08:16.958184 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:08:16.958201 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:08:16.958240 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:08:16.961261 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:08:16.962086 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:08:16.962094 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:08:16.962097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:08:16.962245 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:08:16.962641 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:08:16.962645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:08:16.962647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:08:16.963061 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:08:16.963330 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 03:08:16.963666 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 03:08:16.963883 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 03:08:16.964835 12109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:08:17.054620 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 11977
I20260926 03:08:17.067881 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 12120
Sep 26 03:08:17 dist-test-slave-hmbw krb5kdc[11282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:08:17.083350 12263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:08:17.068673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47144 (local address 127.0.97.3:33341)
0926 03:08:17.068821 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 03:08:17.068826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:08:17.068847 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:08:17.069101 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:08:17.069106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:08:17.069177 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:08:17.069269 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:08:17.070531 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:08:17.071270 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:08:17.072631 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:08:17.073032 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:08:17.073150 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:08:17.077690 (+ 4540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:08:17.077707 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:08:17.077725 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:08:17.077771 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:08:17.080621 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:08:17.081420 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:08:17.081429 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:08:17.081431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:08:17.081582 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:08:17.081969 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:08:17.081972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:08:17.081974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:08:17.082343 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:08:17.082592 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 03:08:17.082890 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 03:08:17.083082 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 03:08:17.084015 12243 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:08:17.172693 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 12120
I20260926 03:08:17.185652 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 11726
I20260926 03:08:17.186810 11788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:08:17.299376 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 11726
I20260926 03:08:17.318184 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 03:08:17.653838 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 03:08:17.654059 388 mini_ranger.cc:66] Stopping Ranger...
I20260926 03:08:17.989979 388 mini_ranger.cc:68] Stopped Ranger
2026-09-26 03:08:17.990 UTC [11312] LOG: received smart shutdown request
2026-09-26 03:08:17.993 UTC [11312] LOG: background worker "logical replication launcher" (PID 11321) exited with exit code 1
2026-09-26 03:08:17.993 UTC [11316] LOG: shutting down
2026-09-26 03:08:18.193 UTC [11312] LOG: database system is shut down
2026-09-26T03:08:18Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50928 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:08:18 dist-test-slave-hmbw krb5kdc[12306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:08:18 dist-test-slave-hmbw krb5kdc[12306](info): set up 2 sockets
Sep 26 03:08:18 dist-test-slave-hmbw krb5kdc[12306](info): commencing operation
krb5kdc: starting...
W20260926 03:08:19.314553 388 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:08:19 dist-test-slave-hmbw krb5kdc[12306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:08:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:08:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260926 03:08:19.638334 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskuHAswW/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/postgres -l logfile start
2026-09-26 03:08:22.403 UTC [12336] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 03:08:22.403 UTC [12336] LOG: listening on IPv4 address "127.0.97.20", port 49881
2026-09-26 03:08:22.407 UTC [12336] LOG: listening on Unix socket "/tmp/.s.PGSQL.49881"
2026-09-26 03:08:22.429 UTC [12339] LOG: database system was shut down at 2026-09-26 03:08:20 UTC
2026-09-26 03:08:22.432 UTC [12336] LOG: database system is ready to accept connections
I20260926 03:08:23.318287 388 mini_postgres.cc:96] Postgres bound to 49881
2026-09-26 03:08:23.334 UTC [12347] FATAL: database "slave" does not exist
127.0.97.20:49881 - accepting connections
I20260926 03:08:23.335179 388 mini_ranger.cc:161] Starting Ranger...
I20260926 03:08:23.365118 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 03:08:23.909157 388 mini_ranger.cc:87] Created ranger Postgres database
I20260926 03:08:23.909277 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ranger-admin
2026-09-26 03:08:24,197 [I] DB FLAVOR :POSTGRES
2026-09-26 03:08:24,198 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 03:08:24,198 [I] Checking connection..
2026-09-26 03:08:24,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 03:08:24,460 [I] Checking connection passed.
2026-09-26 03:08:24,460 [I] --------- Verifying version history table ---------
2026-09-26 03:08:24,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:08:24,709 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 03:08:24,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:08:24,968 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 03:08:24,968 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 03:08:24,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49881/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 03:08:25.230 UTC [12411] WARNING: there is no transaction in progress
2026-09-26 03:08:25,236 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 03:08:25,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:08:25,502 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 03:08:25,502 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 03:08:25,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 03:08:25,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 03:08:25,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hmbw.c.gcp-upstream.internal','N') ;"
2026-09-26 03:08:26,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 03:08:26,483 [I] Table x_portal_user does not exist in database ranger
2026-09-26 03:08:26,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 03:08:26,725 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 03:08:26,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 03:08:26,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 03:08:27,242 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 03:08:27,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49881/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 03:08:28.295 UTC [12519] WARNING: there is no transaction in progress
2026-09-26 03:08:28.313 UTC [12519] WARNING: there is no transaction in progress
2026-09-26 03:08:28.331 UTC [12519] WARNING: there is no transaction in progress
2026-09-26 03:08:28.348 UTC [12519] WARNING: there is no transaction in progress
2026-09-26 03:08:28.366 UTC [12519] WARNING: there is no transaction in progress
2026-09-26 03:08:28.388 UTC [12519] WARNING: there is no transaction in progress
2026-09-26 03:08:28.452 UTC [12519] WARNING: there is no transaction in progress
2026-09-26 03:08:28.460 UTC [12519] WARNING: there is no transaction in progress
2026-09-26 03:08:28.477 UTC [12519] WARNING: there is no transaction in progress
2026-09-26 03:08:28.489 UTC [12519] WARNING: there is no transaction in progress
2026-09-26 03:08:28.500 UTC [12519] WARNING: there is no transaction in progress
2026-09-26 03:08:28.516 UTC [12519] WARNING: there is no transaction in progress
2026-09-26 03:08:28.527 UTC [12519] WARNING: there is no transaction in progress
2026-09-26 03:08:28.534 UTC [12519] WARNING: there is no transaction in progress
2026-09-26 03:08:28.542 UTC [12519] WARNING: there is no transaction in progress
2026-09-26 03:08:28.550 UTC [12519] WARNING: there is no transaction in progress
2026-09-26 03:08:29,333 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 03:08:29,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 03:08:29,569 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 03:08:29,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 03:08:29,832 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 03:08:29,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 03:08:30,082 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 03:08:30,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 03:08:30,332 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 03:08:30,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hmbw.c.gcp-upstream.internal';"
2026-09-26 03:08:30,836 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 03:08:30,837 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 03:08:31,094 [I] DB_PATCHES have already been applied
I20260926 03:08:31.100340 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskuHAswW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 03:08:31.100438 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260926 03:08:31.113991 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:34627
Sep 26, 2026 3:08:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:08:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:08:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 3:08:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 3:08:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 3:08:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 3:08:32 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 03:08:33.240674 388 mini_ranger.cc:160] Time spent starting Ranger: real 9.905s user 0.007s sys 0.045s
Sep 26, 2026 3:08:53 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 3:08:53 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 26, 2026 3:08:53 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 3:08:53 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 3:08:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 3:08:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 3:08:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 3:08:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 3:08:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 3:08:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 3:08:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 3:08:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 3:08:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 3:08:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 3:08:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 3:08:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 3:08:54 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 3:08:54 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 03:08:55.533999 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 03:08:55 dist-test-slave-hmbw krb5kdc[12306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392135, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 03:08:55.718958 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 03:08:55.738567 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 03:08:56.747362 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 03:08:56.747493 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ranger-kms
I20260926 03:08:56.747550 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:49881
2026-09-26 03:08:57,096 [I] DB FLAVOR :POSTGRES
2026-09-26 03:08:57,096 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 03:08:57,096 [I] Checking connection
2026-09-26 03:08:57,096 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49881/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 03:08:57,356 [I] connection success
2026-09-26 03:08:57,357 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 03:08:57,357 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 03:08:57,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49881/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 03:08:57,620 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 03:08:57,620 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 03:08:57,621 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 03:08:57,621 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:49881/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 03:08:57,896 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 03:08:58.438900 388 mini_ranger_kms.cc:310] Created kms service
I20260926 03:08:58.673544 388 mini_ranger_kms.cc:326] Created kudu user
I20260926 03:08:58.721040 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 03:08:58 dist-test-slave-hmbw krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392135, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:08:59.167153 388 mini_ranger_kms.cc:384] Added ranger policy
I20260926 03:08:59.167510 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskuHAswW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskuHAswW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 03:08:59.167562 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260926 03:08:59.181847 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12701
I20260926 03:08:59.181985 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:59421
Sep 26, 2026 3:09:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:09:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:09:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 3:09:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 3:09:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 3:09:00 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 03:09:01.302343 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.583s user 0.073s sys 0.290s
I20260926 03:09:01.302647 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 03:09:01.302834 388 mini_ranger_kms.cc:401] 127.0.97.20:59421/kms/v1/keys
Sep 26 03:09:01 dist-test-slave-hmbw krb5kdc[12306](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790392141, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 03:09:01 dist-test-slave-hmbw krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392141, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 03:09:02 dist-test-slave-hmbw krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392141, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 03:09:04 dist-test-slave-hmbw krb5kdc[12306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:09:04.831624 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40575
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40575
--ranger_config_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59421/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:09:05.180608 12751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:09:05.180932 12751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:09:05.181069 12751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:09:05.188537 12751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:09:05.188683 12751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:09:05.188750 12751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:09:05.188822 12751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:09:05.188885 12751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:09:05.199415 12751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:59421/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40575
--ranger_config_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:09:05.200745 12751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:09:05.202135 12751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:09:05.214437 12756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:05.214412 12759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:05.214352 12757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:05.215353 12751 server_base.cc:1034] running on GCE node
I20260926 03:09:05.216321 12751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:09:05.217955 12751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:09:05.219199 12751 hybrid_clock.cc:648] HybridClock initialized: now 1790392145219143 us; error 55 us; skew 500 ppm
I20260926 03:09:05.219666 12751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:09:05 dist-test-slave-hmbw krb5kdc[12306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:09:05.229602 12751 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:09:05.231366 12751 webserver.cc:460] Webserver started at http://127.0.97.62:40119/ using document root <none> and password file <none>
I20260926 03:09:05.232066 12751 fs_manager.cc:359] Metadata directory not provided
I20260926 03:09:05.232179 12751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:09:05.232465 12751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:09:05 dist-test-slave-hmbw krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:09:05.285615 12751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "cfdf4b81dcd645609d5d20f56d6497a1"
format_stamp: "Formatted at 2026-09-26 03:09:05 on dist-test-slave-hmbw"
server_key: "8ca50d36b13ed44fc48a89cc6517acd5"
server_key_iv: "426d090e53ee93c9faa3dc6a100d3eea"
server_key_version: "kuduclusterkey@0"
I20260926 03:09:05.286674 12751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "cfdf4b81dcd645609d5d20f56d6497a1"
format_stamp: "Formatted at 2026-09-26 03:09:05 on dist-test-slave-hmbw"
server_key: "8ca50d36b13ed44fc48a89cc6517acd5"
server_key_iv: "426d090e53ee93c9faa3dc6a100d3eea"
server_key_version: "kuduclusterkey@0"
I20260926 03:09:05.292682 12751 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:09:05.309878 12766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:05.311480 12751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:09:05.311694 12751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "cfdf4b81dcd645609d5d20f56d6497a1"
format_stamp: "Formatted at 2026-09-26 03:09:05 on dist-test-slave-hmbw"
server_key: "8ca50d36b13ed44fc48a89cc6517acd5"
server_key_iv: "426d090e53ee93c9faa3dc6a100d3eea"
server_key_version: "kuduclusterkey@0"
I20260926 03:09:05.311915 12751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:09:05.334461 12751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:09:05.338876 12751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:09:05.339183 12751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:09:05.358292 12751 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40575
I20260926 03:09:05.358335 12817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40575 every 8 connection(s)
I20260926 03:09:05.359758 12751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:09:05.363938 12818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:09:05.368644 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 12751
I20260926 03:09:05.368875 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:09:05.369807 388 external_mini_cluster.cc:1428] Setting key a68f271c9b14fe65eea0a3e64f3d86ff
I20260926 03:09:05.377827 12818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1: Bootstrap starting.
I20260926 03:09:05.382906 12818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1: Neither blocks nor log segments found. Creating new log.
I20260926 03:09:05.384434 12818 log.cc:824] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1: Log is configured to *not* fsync() on all Append() calls
I20260926 03:09:05.387694 12818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1: No bootstrap required, opened a new log
I20260926 03:09:05.397437 12818 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfdf4b81dcd645609d5d20f56d6497a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40575 } }
I20260926 03:09:05.397923 12818 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:09:05.398038 12818 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfdf4b81dcd645609d5d20f56d6497a1, State: Initialized, Role: FOLLOWER
Sep 26 03:09:05 dist-test-slave-hmbw krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:09:05.398762 12818 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfdf4b81dcd645609d5d20f56d6497a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40575 } }
I20260926 03:09:05.399008 12818 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:09:05.399173 12818 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:09:05.399338 12818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:09:05.402753 12818 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfdf4b81dcd645609d5d20f56d6497a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40575 } }
I20260926 03:09:05.403456 12818 leader_election.cc:302] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfdf4b81dcd645609d5d20f56d6497a1; no voters:
I20260926 03:09:05.404206 12818 leader_election.cc:288] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:09:05.404330 12823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:09:05.411001 12821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:05.372100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46952 (local address 127.0.97.62:40575)
0926 03:09:05.383216 (+ 11116us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:05.383237 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:05.383654 (+ 417us) server_negotiation.cc:407] Connection header received
0926 03:09:05.386311 (+ 2657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:05.386344 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:05.386904 (+ 560us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:05.387379 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:05.388977 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:05.391109 (+ 2132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:05.393074 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:05.394337 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:05.394582 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:05.399658 (+ 5076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:05.399672 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:05.399695 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:05.399751 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:05.403842 (+ 4091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:05.404889 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:05.404901 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:05.404913 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:05.405169 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:05.405661 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:05.405667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:05.405671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:05.406570 (+ 899us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:05.406909 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:05.408766 (+ 1857us) server_negotiation.cc:299] Negotiation successful
0926 03:09:05.409361 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10774,"thread_start_us":10449,"threads_started":1}
I20260926 03:09:05.412406 12823 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [term 1 LEADER]: Becoming Leader. State: Replica: cfdf4b81dcd645609d5d20f56d6497a1, State: Running, Role: LEADER
I20260926 03:09:05.413300 12818 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:09:05.413961 12823 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfdf4b81dcd645609d5d20f56d6497a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40575 } }
I20260926 03:09:05.417708 12824 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfdf4b81dcd645609d5d20f56d6497a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfdf4b81dcd645609d5d20f56d6497a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40575 } } }
I20260926 03:09:05.418287 12824 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [sys.catalog]: This master's current role is: LEADER
I20260926 03:09:05.418952 12825 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfdf4b81dcd645609d5d20f56d6497a1. Latest consensus state: current_term: 1 leader_uuid: "cfdf4b81dcd645609d5d20f56d6497a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfdf4b81dcd645609d5d20f56d6497a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40575 } } }
I20260926 03:09:05.419386 12825 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [sys.catalog]: This master's current role is: LEADER
I20260926 03:09:05.421134 12831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:09:05.427778 12831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:09:05.449028 12831 catalog_manager.cc:1347] Generated new cluster ID: 8fa445acef4a45cea1fc2b041f35d187
I20260926 03:09:05.449159 12831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:09:05.449970 12818 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 03:09:05.485737 12831 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:09:05.487993 12831 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:09:05.501986 12831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1: Generated new TSK 0
I20260926 03:09:05.503338 12831 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:09:06.262696 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40575
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:09:06 dist-test-slave-hmbw krb5kdc[12306](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790392146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 03:09:06.746831 12863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:09:06.747257 12863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:09:06.747462 12863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:09:06.759871 12863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:09:06.760105 12863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:09:06.760365 12863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:09:06.777465 12863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:59421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:09:06.779455 12863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:09:06.781374 12863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:09:06.799242 12873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:06.799866 12874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:06.799988 12876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:06.800302 12863 server_base.cc:1034] running on GCE node
I20260926 03:09:06.800987 12863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:09:06.802135 12863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:09:06.803555 12863 hybrid_clock.cc:648] HybridClock initialized: now 1790392146803470 us; error 78 us; skew 500 ppm
I20260926 03:09:06.804131 12863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:09:06 dist-test-slave-hmbw krb5kdc[12306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:09:06.808164 12863 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:09:06.810020 12863 webserver.cc:460] Webserver started at http://127.0.97.1:35227/ using document root <none> and password file <none>
I20260926 03:09:06.810817 12863 fs_manager.cc:359] Metadata directory not provided
I20260926 03:09:06.810945 12863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:09:06.811295 12863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:09:06 dist-test-slave-hmbw krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:09:06.831974 12863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/data/instance:
uuid: "bf646bb91aac46c5890621f5cf8b6d5b"
format_stamp: "Formatted at 2026-09-26 03:09:06 on dist-test-slave-hmbw"
server_key: "f4f5984c47af0ac65b5e4e6c1334ff7f"
server_key_iv: "f2d9cf1c94007b7f2a13cf946e2f1b7e"
server_key_version: "kuduclusterkey@0"
I20260926 03:09:06.832975 12863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bf646bb91aac46c5890621f5cf8b6d5b"
format_stamp: "Formatted at 2026-09-26 03:09:06 on dist-test-slave-hmbw"
server_key: "f4f5984c47af0ac65b5e4e6c1334ff7f"
server_key_iv: "f2d9cf1c94007b7f2a13cf946e2f1b7e"
server_key_version: "kuduclusterkey@0"
I20260926 03:09:06.851037 12863 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.007s sys 0.013s
I20260926 03:09:06.879469 12883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:06.882261 12863 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.004s sys 0.002s
I20260926 03:09:06.882617 12863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "bf646bb91aac46c5890621f5cf8b6d5b"
format_stamp: "Formatted at 2026-09-26 03:09:06 on dist-test-slave-hmbw"
server_key: "f4f5984c47af0ac65b5e4e6c1334ff7f"
server_key_iv: "f2d9cf1c94007b7f2a13cf946e2f1b7e"
server_key_version: "kuduclusterkey@0"
I20260926 03:09:06.883024 12863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:09:06.911305 12863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:09:06.917733 12863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:09:06.918174 12863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:09:06.925830 12863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:09:06.934392 12863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:09:06.934641 12863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 03:09:06.934877 12863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:09:06.934990 12863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:09:07 dist-test-slave-hmbw krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:09:07.148154 12995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43667 every 8 connection(s)
I20260926 03:09:07.148510 12863 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43667
I20260926 03:09:07.150183 12863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:09:07.159785 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 12863
I20260926 03:09:07.160032 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:09:07.160775 388 external_mini_cluster.cc:1428] Setting key dedfb2666d8520ec71746446391ed555
Sep 26 03:09:07 dist-test-slave-hmbw krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:09:07.208339 12998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:07.159804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40575 (local address 127.0.97.1:49655)
0926 03:09:07.183871 (+ 24067us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:07.183924 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:07.186009 (+ 2085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:07.188283 (+ 2274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:07.188293 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:07.188841 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:09:07.189514 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:07.189543 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:07.190896 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:07.190901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:07.193033 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:07.193052 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:07.194354 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:07.194362 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:07.194702 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:07.195539 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:09:07.195573 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:09:07.198545 (+ 2972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:09:07.203016 (+ 4471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:07.203028 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:07.203051 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:07.203746 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:07.204272 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:07.204277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:07.204281 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:07.204560 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:07.205387 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:09:07.205399 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:09:07.205876 (+ 477us) client_negotiation.cc:769] Sending connection context
0926 03:09:07.206245 (+ 369us) client_negotiation.cc:240] Negotiation successful
0926 03:09:07.206886 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23732,"thread_start_us":26690,"threads_started":1}
I20260926 03:09:07.209689 12997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:07.159276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49655 (local address 127.0.97.62:40575)
0926 03:09:07.187684 (+ 28408us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:07.187689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:07.187708 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:09:07.187766 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:07.187770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:07.187846 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:07.187975 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:07.189761 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:07.190701 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:07.193267 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:07.193883 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:07.194017 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:07.198918 (+ 4901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:07.198926 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:07.198931 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:07.198996 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:07.202742 (+ 3746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:07.203934 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:07.203941 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:07.203944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:07.204128 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:07.204719 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:07.204725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:07.204728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:07.205139 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:07.205287 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:07.208979 (+ 3692us) server_negotiation.cc:299] Negotiation successful
0926 03:09:07.209275 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28285,"thread_start_us":28845,"threads_started":1}
I20260926 03:09:07.212239 12996 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40575
I20260926 03:09:07.212632 12996 heartbeater.cc:464] Registering TS with master...
I20260926 03:09:07.213617 12996 heartbeater.cc:511] Master 127.0.97.62:40575 requested a full tablet report, sending...
I20260926 03:09:07.217016 12783 ts_manager.cc:194] Registered new tserver with Master: bf646bb91aac46c5890621f5cf8b6d5b (127.0.97.1:43667)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 03:09:07.219550 12783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:09:07.258889 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40575
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:09:07 dist-test-slave-hmbw krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260926 03:09:07.763065 13002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:09:07.763381 13002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:09:07.763477 13002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:09:07.771915 13002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:09:07.772078 13002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:09:07.772198 13002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:09:07.783128 13002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:59421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:09:07.784442 13002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:09:07.785742 13002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:09:07.800575 13016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:07.800743 13019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:07.800549 13017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:07.801421 13002 server_base.cc:1034] running on GCE node
I20260926 03:09:07.802075 13002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:09:07.803195 13002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:09:07.804438 13002 hybrid_clock.cc:648] HybridClock initialized: now 1790392147804377 us; error 54 us; skew 500 ppm
I20260926 03:09:07.804999 13002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:09:07 dist-test-slave-hmbw krb5kdc[12306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:09:07.808655 13002 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:09:07.812912 13002 webserver.cc:460] Webserver started at http://127.0.97.2:34941/ using document root <none> and password file <none>
I20260926 03:09:07.813932 13002 fs_manager.cc:359] Metadata directory not provided
I20260926 03:09:07.814111 13002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:09:07.814468 13002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:09:07 dist-test-slave-hmbw krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:09:07.831137 13002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/data/instance:
uuid: "44fdbdfab23f44d09972b372222da579"
format_stamp: "Formatted at 2026-09-26 03:09:07 on dist-test-slave-hmbw"
server_key: "a7796cfefbfb379e1e753b0792c8a74d"
server_key_iv: "7f16e1431e5cb1a093b31d5f7dce021f"
server_key_version: "kuduclusterkey@0"
I20260926 03:09:07.832048 13002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/wal/instance:
uuid: "44fdbdfab23f44d09972b372222da579"
format_stamp: "Formatted at 2026-09-26 03:09:07 on dist-test-slave-hmbw"
server_key: "a7796cfefbfb379e1e753b0792c8a74d"
server_key_iv: "7f16e1431e5cb1a093b31d5f7dce021f"
server_key_version: "kuduclusterkey@0"
I20260926 03:09:07.837973 13002 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 03:09:07.855111 13026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:07.856720 13002 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260926 03:09:07.856904 13002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "44fdbdfab23f44d09972b372222da579"
format_stamp: "Formatted at 2026-09-26 03:09:07 on dist-test-slave-hmbw"
server_key: "a7796cfefbfb379e1e753b0792c8a74d"
server_key_iv: "7f16e1431e5cb1a093b31d5f7dce021f"
server_key_version: "kuduclusterkey@0"
I20260926 03:09:07.857138 13002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:09:07.895362 13002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:09:07.900689 13002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:09:07.901012 13002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:09:07.902014 13002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:09:07.903517 13002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:09:07.903595 13002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:07.903713 13002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:09:07.903761 13002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:07.937330 13002 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37055
I20260926 03:09:07.937359 13138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37055 every 8 connection(s)
I20260926 03:09:07.938798 13002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:09:07.940239 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 13002
I20260926 03:09:07.940373 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:09:07.940752 388 external_mini_cluster.cc:1428] Setting key 8d5346d4d1d11db4345f112db8e28d67
Sep 26 03:09:07 dist-test-slave-hmbw krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 03:09:07.968818 13141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:07.942779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39021 (local address 127.0.97.62:40575)
0926 03:09:07.950984 (+ 8205us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:07.950989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:07.951011 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:09:07.951072 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:07.951076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:07.951146 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:07.951291 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:07.953626 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:07.954378 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:07.956936 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:07.957426 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:07.957555 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:07.962011 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:07.962018 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:07.962021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:07.962061 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:07.964722 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:07.966031 (+ 1309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:07.966036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:07.966038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:07.966207 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:07.966907 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:07.966912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:07.966914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:07.967225 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:07.967349 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:07.968300 (+ 951us) server_negotiation.cc:299] Negotiation successful
0926 03:09:07.968500 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8048,"thread_start_us":7939,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 03:09:07.969898 13142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:07.943342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40575 (local address 127.0.97.2:39021)
0926 03:09:07.944225 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:07.944271 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:07.946082 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:07.951650 (+ 5568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:07.951662 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:07.952266 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:09:07.953318 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:07.953352 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:07.954578 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:07.954584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:07.956672 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:07.956695 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:07.957719 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:07.957728 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:07.958051 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:07.958929 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:09:07.958965 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:09:07.961675 (+ 2710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:09:07.964923 (+ 3248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:07.964934 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:07.964956 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:07.965842 (+ 886us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:07.966348 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:07.966354 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:07.966358 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:07.966762 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:07.967435 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:09:07.967442 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:09:07.967904 (+ 462us) client_negotiation.cc:769] Sending connection context
0926 03:09:07.968173 (+ 269us) client_negotiation.cc:240] Negotiation successful
0926 03:09:07.968604 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":402,"threads_started":1}
I20260926 03:09:07.971531 13139 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40575
I20260926 03:09:07.971872 13139 heartbeater.cc:464] Registering TS with master...
I20260926 03:09:07.972764 13139 heartbeater.cc:511] Master 127.0.97.62:40575 requested a full tablet report, sending...
I20260926 03:09:07.974756 12783 ts_manager.cc:194] Registered new tserver with Master: 44fdbdfab23f44d09972b372222da579 (127.0.97.2:37055)
I20260926 03:09:07.975855 12783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:09:07.999941 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40575
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:09:08.224011 12996 heartbeater.cc:503] Master 127.0.97.62:40575 was elected leader, sending a full tablet report...
W20260926 03:09:08.354753 13145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:09:08.355057 13145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:09:08.355178 13145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:09:08.365895 13145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:09:08.366060 13145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:09:08.366236 13145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:09:08.378269 13145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:59421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:09:08.379724 13145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:09:08.381080 13145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:09:08.395203 13150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:08.395203 13151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:08.395316 13153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:08.396600 13145 server_base.cc:1034] running on GCE node
I20260926 03:09:08.397305 13145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:09:08.398375 13145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:09:08.399631 13145 hybrid_clock.cc:648] HybridClock initialized: now 1790392148399592 us; error 73 us; skew 500 ppm
I20260926 03:09:08.399993 13145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:09:08 dist-test-slave-hmbw krb5kdc[12306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:09:08.403432 13145 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:09:08.405138 13145 webserver.cc:460] Webserver started at http://127.0.97.3:37569/ using document root <none> and password file <none>
I20260926 03:09:08.405860 13145 fs_manager.cc:359] Metadata directory not provided
I20260926 03:09:08.405972 13145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:09:08.406255 13145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:09:08 dist-test-slave-hmbw krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:09:08.423166 13145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/data/instance:
uuid: "6d4d384eecb34663ab6522e0392e065f"
format_stamp: "Formatted at 2026-09-26 03:09:08 on dist-test-slave-hmbw"
server_key: "51776364ac71b6a0f4886ed6e74620ae"
server_key_iv: "8ec67348ae77855327ed1a2cfbdffedb"
server_key_version: "kuduclusterkey@0"
I20260926 03:09:08.424047 13145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6d4d384eecb34663ab6522e0392e065f"
format_stamp: "Formatted at 2026-09-26 03:09:08 on dist-test-slave-hmbw"
server_key: "51776364ac71b6a0f4886ed6e74620ae"
server_key_iv: "8ec67348ae77855327ed1a2cfbdffedb"
server_key_version: "kuduclusterkey@0"
I20260926 03:09:08.430181 13145 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 03:09:08.449541 13160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:08.451723 13145 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 03:09:08.451964 13145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "6d4d384eecb34663ab6522e0392e065f"
format_stamp: "Formatted at 2026-09-26 03:09:08 on dist-test-slave-hmbw"
server_key: "51776364ac71b6a0f4886ed6e74620ae"
server_key_iv: "8ec67348ae77855327ed1a2cfbdffedb"
server_key_version: "kuduclusterkey@0"
I20260926 03:09:08.452260 13145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:09:08.475091 13145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:09:08.479887 13145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:09:08.480168 13145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:09:08.481168 13145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:09:08.482826 13145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:09:08.482910 13145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:08.483004 13145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:09:08.483052 13145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:08.518312 13145 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44205
I20260926 03:09:08.518350 13272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44205 every 8 connection(s)
I20260926 03:09:08.519649 13145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:09:08.525272 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 13145
I20260926 03:09:08.525429 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:09:08.525926 388 external_mini_cluster.cc:1428] Setting key 7b5d494e865b9c8adea244fccd6c0a84
Sep 26 03:09:08 dist-test-slave-hmbw krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:09:08.543064 13275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:08.522447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:32789 (local address 127.0.97.62:40575)
0926 03:09:08.523008 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:08.523013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:08.524272 (+ 1259us) server_negotiation.cc:407] Connection header received
0926 03:09:08.526219 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:08.526225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:08.526318 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:08.526411 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:08.528229 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:08.529284 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:08.531099 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:08.531548 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:08.531663 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:08.535597 (+ 3934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:08.535603 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:08.535607 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:08.535653 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:08.538376 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:08.539358 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:08.539363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:08.539365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:08.539545 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:08.540257 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:08.540265 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:08.540267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:08.540801 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:08.541057 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:08.542158 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0926 03:09:08.542695 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":335,"threads_started":1}
I20260926 03:09:08.544237 13276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:08.522993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40575 (local address 127.0.97.3:32789)
0926 03:09:08.524071 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:08.524127 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:08.525875 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:08.526778 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:08.526792 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:08.527334 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:09:08.528015 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:08.528045 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:08.529512 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:08.529517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:08.530917 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:08.530932 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:08.531811 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:08.531817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:08.532051 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:08.533019 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:09:08.533043 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:09:08.535294 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:09:08.538630 (+ 3336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:08.538641 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:08.538661 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:08.539188 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:08.539715 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:08.539719 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:08.539722 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:08.539951 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:08.541106 (+ 1155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:09:08.541118 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:09:08.541642 (+ 524us) client_negotiation.cc:769] Sending connection context
0926 03:09:08.541997 (+ 355us) client_negotiation.cc:240] Negotiation successful
0926 03:09:08.542889 (+ 892us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":308,"threads_started":1}
I20260926 03:09:08.546209 13273 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40575
I20260926 03:09:08.546702 13273 heartbeater.cc:464] Registering TS with master...
I20260926 03:09:08.547751 13273 heartbeater.cc:511] Master 127.0.97.62:40575 requested a full tablet report, sending...
I20260926 03:09:08.549841 12783 ts_manager.cc:194] Registered new tserver with Master: 6d4d384eecb34663ab6522e0392e065f (127.0.97.3:44205)
I20260926 03:09:08.550783 12783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:32789
I20260926 03:09:08.554967 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:09:08.556710 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 12863
Sep 26 03:09:08 dist-test-slave-hmbw krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:09:08.574116 13278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:08.557773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42326 (local address 127.0.97.1:43667)
0926 03:09:08.558413 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:08.558423 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:08.558452 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:09:08.558559 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:08.558569 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:08.558799 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:08.559069 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:08.560472 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:08.561517 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:08.563064 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:08.563648 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:08.563771 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:08.568065 (+ 4294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:08.568080 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:08.568096 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:08.568139 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:08.571115 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:08.571962 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:08.571970 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:08.571973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:08.572137 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:08.572522 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:08.572527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:08.572529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:08.572876 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:08.573190 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:08.573489 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 03:09:08.573761 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":300,"threads_started":1}
I20260926 03:09:08.575496 12988 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:09:08.661899 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 12863
I20260926 03:09:08.674892 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 13002
Sep 26 03:09:08 dist-test-slave-hmbw krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:09:08.692644 13280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:08.675784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52856 (local address 127.0.97.2:37055)
0926 03:09:08.676312 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:08.676319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:08.676346 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:09:08.676406 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:08.676413 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:08.676632 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:08.676767 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:08.678083 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:08.679340 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:08.680872 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:08.681437 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:08.681538 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:08.686036 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:08.686054 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:08.686070 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:08.686115 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:08.689287 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:08.690188 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:08.690197 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:08.690200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:08.690361 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:08.690964 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:08.690968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:08.690970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:08.691343 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:08.691640 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:08.691982 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 03:09:08.692265 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":276,"threads_started":1}
I20260926 03:09:08.694865 13134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:09:08.785152 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 13002
I20260926 03:09:08.800786 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 13145
Sep 26 03:09:08 dist-test-slave-hmbw krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:09:08.819258 13282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:08.801830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41280 (local address 127.0.97.3:44205)
0926 03:09:08.802486 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:08.802517 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:08.802560 (+ 43us) server_negotiation.cc:407] Connection header received
0926 03:09:08.802631 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:08.802639 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:08.802883 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:08.803050 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:08.804347 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:08.805655 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:08.807366 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:08.808064 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:08.808198 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:08.812888 (+ 4690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:08.812906 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:08.812922 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:08.812972 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:08.816078 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:08.816972 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:08.816980 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:08.816982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:08.817134 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:08.817557 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:08.817561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:08.817563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:08.817942 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:08.818259 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:08.818618 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 03:09:08.818888 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":341,"threads_started":1}
I20260926 03:09:08.820837 13268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:09:08.906414 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 13145
I20260926 03:09:08.920430 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 12751
I20260926 03:09:08.921686 12813 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:09:09.038276 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 12751
I20260926 03:09:09.058308 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40575
--webserver_interface=127.0.97.62
--webserver_port=40119
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40575
--ranger_config_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59421/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59421/kms with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:09:09.396695 13285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:09:09.397009 13285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:09:09.397125 13285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:09:09.404677 13285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:09:09.404809 13285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:09:09.404865 13285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:09:09.404942 13285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:09:09.405005 13285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:09:09.415812 13285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59421/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40575
--ranger_config_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=40119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:09:09.417461 13285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:09:09.418898 13285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:09:09.431526 13290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:09.431463 13291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:09.431414 13293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:09.432402 13285 server_base.cc:1034] running on GCE node
I20260926 03:09:09.433250 13285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:09:09.434826 13285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:09:09.436102 13285 hybrid_clock.cc:648] HybridClock initialized: now 1790392149436042 us; error 49 us; skew 500 ppm
I20260926 03:09:09.436478 13285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:09:09 dist-test-slave-hmbw krb5kdc[12306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:09:09.439945 13285 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:09:09.441576 13285 webserver.cc:460] Webserver started at http://127.0.97.62:40119/ using document root <none> and password file <none>
I20260926 03:09:09.442271 13285 fs_manager.cc:359] Metadata directory not provided
I20260926 03:09:09.442399 13285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 03:09:09 dist-test-slave-hmbw krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:09:09.506556 13285 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:09:09.510030 13301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:09.511971 13285 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260926 03:09:09.512199 13285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "cfdf4b81dcd645609d5d20f56d6497a1"
format_stamp: "Formatted at 2026-09-26 03:09:05 on dist-test-slave-hmbw"
server_key: "8ca50d36b13ed44fc48a89cc6517acd5"
server_key_iv: "426d090e53ee93c9faa3dc6a100d3eea"
server_key_version: "kuduclusterkey@0"
I20260926 03:09:09.513324 13285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:09:09.537961 13285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:09:09.542665 13285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:09:09.542969 13285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:09:09.562038 13285 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40575
I20260926 03:09:09.561976 13352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40575 every 8 connection(s)
I20260926 03:09:09.563922 13285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:09:09.572088 13353 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:09:09.573298 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 13285
I20260926 03:09:09.574146 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43667
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:40575
--webserver_port=35227
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59421/kms with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:09:09.576894 13353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1: Bootstrap starting.
I20260926 03:09:09.598333 13353 log.cc:824] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1: Log is configured to *not* fsync() on all Append() calls
I20260926 03:09:09.608481 13353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:09:09.609140 13353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1: Bootstrap complete.
I20260926 03:09:09.618765 13353 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfdf4b81dcd645609d5d20f56d6497a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40575 } }
I20260926 03:09:09.619725 13353 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfdf4b81dcd645609d5d20f56d6497a1, State: Initialized, Role: FOLLOWER
I20260926 03:09:09.620371 13353 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfdf4b81dcd645609d5d20f56d6497a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40575 } }
I20260926 03:09:09.620683 13353 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:09:09.620806 13353 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:09:09.620975 13353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:09:09.624991 13353 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfdf4b81dcd645609d5d20f56d6497a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40575 } }
I20260926 03:09:09.625602 13353 leader_election.cc:302] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfdf4b81dcd645609d5d20f56d6497a1; no voters:
I20260926 03:09:09.626206 13353 leader_election.cc:288] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 03:09:09.626351 13357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 03:09:09.628561 13353 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:09:09.633441 13357 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [term 2 LEADER]: Becoming Leader. State: Replica: cfdf4b81dcd645609d5d20f56d6497a1, State: Running, Role: LEADER
I20260926 03:09:09.634387 13357 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfdf4b81dcd645609d5d20f56d6497a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40575 } }
I20260926 03:09:09.637223 13358 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cfdf4b81dcd645609d5d20f56d6497a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfdf4b81dcd645609d5d20f56d6497a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40575 } } }
I20260926 03:09:09.637661 13358 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [sys.catalog]: This master's current role is: LEADER
I20260926 03:09:09.638211 13359 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfdf4b81dcd645609d5d20f56d6497a1. Latest consensus state: current_term: 2 leader_uuid: "cfdf4b81dcd645609d5d20f56d6497a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfdf4b81dcd645609d5d20f56d6497a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40575 } } }
I20260926 03:09:09.638490 13359 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1 [sys.catalog]: This master's current role is: LEADER
I20260926 03:09:09.640703 13369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:09:09.656035 13369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:09:09.657809 13369 catalog_manager.cc:1259] Loaded cluster ID: 8fa445acef4a45cea1fc2b041f35d187
I20260926 03:09:09.658028 13369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:09:09.662899 13369 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:09:09.664959 13369 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1: Loaded TSK: 0
I20260926 03:09:09.667080 13369 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:09:09.669836 13353 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260926 03:09:10.082058 13355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:09:10.082540 13355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:09:10.082773 13355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:09:10.095764 13355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:09:10.096005 13355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:09:10.096233 13355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:09:10.112828 13355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43667
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=35227
--tserver_master_addrs=127.0.97.62:40575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:09:10.114924 13355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:09:10.116894 13355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:09:10.168421 13392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:10.168929 13391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:10.179896 13394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:10.180573 13355 server_base.cc:1034] running on GCE node
I20260926 03:09:10.181677 13355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:09:10.189020 13355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:09:10.192674 13355 hybrid_clock.cc:648] HybridClock initialized: now 1790392150192575 us; error 83 us; skew 500 ppm
I20260926 03:09:10.193301 13355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:09:10 dist-test-slave-hmbw krb5kdc[12306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:09:10.198590 13355 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:09:10.204674 13355 webserver.cc:460] Webserver started at http://127.0.97.1:35227/ using document root <none> and password file <none>
I20260926 03:09:10.205767 13355 fs_manager.cc:359] Metadata directory not provided
I20260926 03:09:10.205958 13355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 03:09:10 dist-test-slave-hmbw krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:09:10.264851 13355 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.003s sys 0.008s
I20260926 03:09:10.278431 13403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:10.280727 13355 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.007s sys 0.005s
I20260926 03:09:10.280951 13355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "bf646bb91aac46c5890621f5cf8b6d5b"
format_stamp: "Formatted at 2026-09-26 03:09:06 on dist-test-slave-hmbw"
server_key: "f4f5984c47af0ac65b5e4e6c1334ff7f"
server_key_iv: "f2d9cf1c94007b7f2a13cf946e2f1b7e"
server_key_version: "kuduclusterkey@0"
I20260926 03:09:10.282429 13355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:09:10.331797 13355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:09:10.336521 13355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:09:10.337146 13355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:09:10.339085 13355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:09:10.341180 13355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:09:10.341318 13355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:10.341459 13355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:09:10.341547 13355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:10.407253 13355 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43667
I20260926 03:09:10.407317 13515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43667 every 8 connection(s)
I20260926 03:09:10.408978 13355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:09:10.419314 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 13355
I20260926 03:09:10.420150 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37055
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:40575
--webserver_port=34941
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59421/kms with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:09:10 dist-test-slave-hmbw krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:09:10.449306 13518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:10.412229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41607 (local address 127.0.97.62:40575)
0926 03:09:10.413057 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:10.413077 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:10.413714 (+ 637us) server_negotiation.cc:407] Connection header received
0926 03:09:10.417460 (+ 3746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:10.417480 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:10.418042 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:10.418484 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:10.421639 (+ 3155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:10.424794 (+ 3155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:10.427714 (+ 2920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:10.428948 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:10.429182 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:10.435915 (+ 6733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:10.435929 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:10.435951 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:10.436015 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:10.440174 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:10.441446 (+ 1272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:10.441459 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:10.441468 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:10.441741 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:10.442411 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:10.442417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:10.442420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:10.443301 (+ 881us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:10.443518 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:10.447115 (+ 3597us) server_negotiation.cc:299] Negotiation successful
0926 03:09:10.447767 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":332,"threads_started":1}
I20260926 03:09:10.449823 13519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:10.412250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40575 (local address 127.0.97.1:41607)
0926 03:09:10.413500 (+ 1250us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:10.413551 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:10.416683 (+ 3132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:10.419517 (+ 2834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:10.419529 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:10.420209 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:09:10.421335 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:10.421371 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:10.425120 (+ 3749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:10.425126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:10.427436 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:10.427461 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:10.429672 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:10.429683 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:10.429992 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:10.432682 (+ 2690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:09:10.432722 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:09:10.435487 (+ 2765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:09:10.440451 (+ 4964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:10.440463 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:10.440482 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:10.441202 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:10.441928 (+ 726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:10.441934 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:10.441938 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:10.442228 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:10.443610 (+ 1382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:09:10.443619 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:09:10.444076 (+ 457us) client_negotiation.cc:769] Sending connection context
0926 03:09:10.444411 (+ 335us) client_negotiation.cc:240] Negotiation successful
0926 03:09:10.445069 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"thread_start_us":751,"threads_started":1}
I20260926 03:09:10.455060 13516 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40575
I20260926 03:09:10.455467 13516 heartbeater.cc:464] Registering TS with master...
I20260926 03:09:10.456669 13516 heartbeater.cc:511] Master 127.0.97.62:40575 requested a full tablet report, sending...
W20260926 03:09:10.460721 13516 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40575 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 03:09:10.460989 13516 heartbeater.cc:464] Registering TS with master...
I20260926 03:09:10.461310 13516 heartbeater.cc:511] Master 127.0.97.62:40575 requested a full tablet report, sending...
I20260926 03:09:10.462842 13516 heartbeater.cc:464] Registering TS with master...
I20260926 03:09:10.463145 13516 heartbeater.cc:511] Master 127.0.97.62:40575 requested a full tablet report, sending...
W20260926 03:09:10.464308 13516 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 03:09:10.915340 13520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:09:10.915863 13520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:09:10.916081 13520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:09:10.927525 13520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:09:10.927753 13520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:09:10.927965 13520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:09:10.946218 13520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34941
--tserver_master_addrs=127.0.97.62:40575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:09:10.948040 13520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:09:10.949795 13520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:09:10.970099 13536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:10.970075 13537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:10.976032 13539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:10.976382 13520 server_base.cc:1034] running on GCE node
I20260926 03:09:10.977268 13520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:09:10.978381 13520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:09:10.979652 13520 hybrid_clock.cc:648] HybridClock initialized: now 1790392150979595 us; error 54 us; skew 500 ppm
I20260926 03:09:10.980011 13520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:09:10 dist-test-slave-hmbw krb5kdc[12306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:09:10.983381 13520 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:09:10.988271 13520 webserver.cc:460] Webserver started at http://127.0.97.2:34941/ using document root <none> and password file <none>
I20260926 03:09:10.989182 13520 fs_manager.cc:359] Metadata directory not provided
I20260926 03:09:10.989351 13520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 03:09:10 dist-test-slave-hmbw krb5kdc[12306](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790392150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 03:09:10 dist-test-slave-hmbw krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:09:11.023602 13520 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 03:09:11.033815 13547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:11.035614 13520 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.004s sys 0.006s
I20260926 03:09:11.036481 13520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "44fdbdfab23f44d09972b372222da579"
format_stamp: "Formatted at 2026-09-26 03:09:07 on dist-test-slave-hmbw"
server_key: "a7796cfefbfb379e1e753b0792c8a74d"
server_key_iv: "7f16e1431e5cb1a093b31d5f7dce021f"
server_key_version: "kuduclusterkey@0"
I20260926 03:09:11.037734 13520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:09:11.067229 13520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:09:11.073256 13520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:09:11.073612 13520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:09:11.074847 13520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:09:11.076469 13520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:09:11.076570 13520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:11.076668 13520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:09:11.076735 13520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:11.132395 13520 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37055
I20260926 03:09:11.132583 13660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37055 every 8 connection(s)
I20260926 03:09:11.133940 13520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:09:11.139807 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 13520
I20260926 03:09:11.140604 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44205
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:40575
--webserver_port=37569
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59421/kms with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:09:11 dist-test-slave-hmbw krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:09:11.163177 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:11.137396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40957 (local address 127.0.97.62:40575)
0926 03:09:11.137997 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:11.138004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:11.139219 (+ 1215us) server_negotiation.cc:407] Connection header received
0926 03:09:11.142656 (+ 3437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:11.142663 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:11.142775 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:11.142880 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:11.145604 (+ 2724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:11.146714 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:11.149338 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:11.149983 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:11.150116 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:11.155034 (+ 4918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:11.155041 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:11.155046 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:11.155094 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:11.158844 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:11.160055 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:11.160061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:11.160065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:11.160234 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:11.160883 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:11.160889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:11.160892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:11.161284 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:11.161423 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:11.162555 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0926 03:09:11.162817 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":303,"threads_started":1}
I20260926 03:09:11.164732 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:11.138045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40575 (local address 127.0.97.2:40957)
0926 03:09:11.139001 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:11.139059 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:11.142131 (+ 3072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:11.143627 (+ 1496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:11.143643 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:11.144315 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:09:11.145285 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:11.145320 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:11.146942 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:11.146948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:11.149110 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:11.149143 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:11.150281 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:11.150291 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:11.150668 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:11.151536 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:09:11.151579 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:09:11.154644 (+ 3065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:09:11.159110 (+ 4466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:11.159122 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:11.159147 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:11.159864 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:11.160415 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:11.160420 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:11.160423 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:11.160725 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:11.161558 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:09:11.161568 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:09:11.162029 (+ 461us) client_negotiation.cc:769] Sending connection context
0926 03:09:11.162368 (+ 339us) client_negotiation.cc:240] Negotiation successful
0926 03:09:11.163026 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":386,"threads_started":1}
I20260926 03:09:11.168586 13661 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40575
I20260926 03:09:11.169117 13661 heartbeater.cc:464] Registering TS with master...
I20260926 03:09:11.170257 13661 heartbeater.cc:511] Master 127.0.97.62:40575 requested a full tablet report, sending...
I20260926 03:09:11.173313 13318 ts_manager.cc:194] Registered new tserver with Master: 44fdbdfab23f44d09972b372222da579 (127.0.97.2:37055)
I20260926 03:09:11.175845 13318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40957
I20260926 03:09:11.464468 13516 heartbeater.cc:464] Registering TS with master...
I20260926 03:09:11.464898 13516 heartbeater.cc:511] Master 127.0.97.62:40575 requested a full tablet report, sending...
I20260926 03:09:11.466104 13318 ts_manager.cc:194] Registered new tserver with Master: bf646bb91aac46c5890621f5cf8b6d5b (127.0.97.1:43667)
I20260926 03:09:11.467417 13318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41607
Sep 26 03:09:11 dist-test-slave-hmbw krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260926 03:09:11.583112 13665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:09:11.583425 13665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:09:11.583600 13665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:09:11.591750 13665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:09:11.591907 13665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:09:11.592111 13665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:09:11.603044 13665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44205
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=37569
--tserver_master_addrs=127.0.97.62:40575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:09:11.604565 13665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:09:11.605899 13665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:09:11.627162 13671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:11.628648 13673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:11.629244 13670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:11.630376 13665 server_base.cc:1034] running on GCE node
I20260926 03:09:11.631222 13665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:09:11.632500 13665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:09:11.633792 13665 hybrid_clock.cc:648] HybridClock initialized: now 1790392151633732 us; error 49 us; skew 500 ppm
I20260926 03:09:11.634218 13665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:09:11 dist-test-slave-hmbw krb5kdc[12306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:09:11.639603 13665 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:09:11.644186 13665 webserver.cc:460] Webserver started at http://127.0.97.3:37569/ using document root <none> and password file <none>
I20260926 03:09:11.645300 13665 fs_manager.cc:359] Metadata directory not provided
I20260926 03:09:11.645483 13665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 03:09:11 dist-test-slave-hmbw krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:09:11.684161 13665 fs_manager.cc:709] Time spent opening directory manager: real 0.017s user 0.006s sys 0.009s
I20260926 03:09:11.688388 13681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:11.690460 13665 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260926 03:09:11.690701 13665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "6d4d384eecb34663ab6522e0392e065f"
format_stamp: "Formatted at 2026-09-26 03:09:08 on dist-test-slave-hmbw"
server_key: "51776364ac71b6a0f4886ed6e74620ae"
server_key_iv: "8ec67348ae77855327ed1a2cfbdffedb"
server_key_version: "kuduclusterkey@0"
I20260926 03:09:11.692162 13665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:09:11.728367 13665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:09:11.733776 13665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:09:11.734172 13665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:09:11.737949 13665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:09:11.740166 13665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:09:11.740360 13665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 03:09:11.740509 13665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:09:11.740612 13665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:09:11 dist-test-slave-hmbw krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:09:12.042989 13665 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44205
I20260926 03:09:12.043046 13794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44205 every 8 connection(s)
I20260926 03:09:12.045171 13665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:09:12.045930 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 13665
I20260926 03:09:12.087226 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:12.047031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46960 (local address 127.0.97.62:40575)
0926 03:09:12.062333 (+ 15302us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:12.062338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:12.062360 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:09:12.062423 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:12.062427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:12.062556 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:12.062717 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:12.064326 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.065064 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.074944 (+ 9880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.075458 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.075695 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:12.079626 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:12.079632 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:12.079637 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:12.079726 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:12.084035 (+ 4309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:12.085023 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:12.085029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:12.085032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:12.085180 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:12.085602 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:12.085605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:12.085607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:12.085863 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:12.086052 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:12.086581 (+ 529us) server_negotiation.cc:299] Negotiation successful
0926 03:09:12.086822 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15171,"thread_start_us":15520,"threads_started":1}
Sep 26 03:09:12 dist-test-slave-hmbw krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:09:12.096817 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:12.068007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57777 (local address 127.0.97.62:40575)
0926 03:09:12.079987 (+ 11980us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:12.079993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:12.080134 (+ 141us) server_negotiation.cc:407] Connection header received
0926 03:09:12.080206 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:12.080211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:12.080320 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:12.080555 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:12.082632 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.084019 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.086375 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.087061 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.087181 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:12.090870 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:12.090875 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:12.090879 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:12.090914 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:12.093365 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:12.094299 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:12.094304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:12.094306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:12.094465 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:12.095009 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:12.095013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:12.095015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:12.095304 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:12.095435 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:12.096304 (+ 869us) server_negotiation.cc:299] Negotiation successful
0926 03:09:12.096532 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11842,"thread_start_us":12627,"threads_started":1}
I20260926 03:09:12.097769 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:12.067379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40575 (local address 127.0.97.3:57777)
0926 03:09:12.068320 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:12.068383 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:12.071436 (+ 3053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:12.081000 (+ 9564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:12.081019 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:12.081705 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:09:12.082372 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:12.082395 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.084266 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.084272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:12.086209 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:12.086228 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.087300 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.087308 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:12.087647 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:12.088339 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:09:12.088377 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:09:12.090627 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:09:12.093558 (+ 2931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:12.093567 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:12.093584 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:12.094094 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:12.094657 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:12.094661 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:12.094663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:12.094893 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:12.095471 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:09:12.095479 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:09:12.095869 (+ 390us) client_negotiation.cc:769] Sending connection context
0926 03:09:12.096143 (+ 274us) client_negotiation.cc:240] Negotiation successful
0926 03:09:12.096591 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":689,"threads_started":1}
I20260926 03:09:12.099319 13797 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40575
I20260926 03:09:12.099610 13797 heartbeater.cc:464] Registering TS with master...
I20260926 03:09:12.100346 13797 heartbeater.cc:511] Master 127.0.97.62:40575 requested a full tablet report, sending...
I20260926 03:09:12.101943 13318 ts_manager.cc:194] Registered new tserver with Master: 6d4d384eecb34663ab6522e0392e065f (127.0.97.3:44205)
I20260926 03:09:12.102584 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:09:12.103113 13318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57777
I20260926 03:09:12.119488 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:12.105933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46970 (local address 127.0.97.62:40575)
0926 03:09:12.106128 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:12.106133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:12.106479 (+ 346us) server_negotiation.cc:407] Connection header received
0926 03:09:12.106656 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:12.106660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:12.106723 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:12.106809 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:12.108283 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.109013 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.110450 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.110924 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.111036 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:12.113714 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:12.113720 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:12.113723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:12.113764 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:12.116473 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:12.117299 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:12.117304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:12.117306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:12.117454 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:12.117991 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:12.117995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:12.117997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:12.118306 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:12.118564 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:12.118923 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 03:09:12.119158 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 03:09:12.125388 13318 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:09:12.133196 13318 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:09:12.161645 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:12.149832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41288 (local address 127.0.97.3:44205)
0926 03:09:12.150577 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:12.150587 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:12.152820 (+ 2233us) server_negotiation.cc:407] Connection header received
0926 03:09:12.152937 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:12.152946 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:12.153217 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:12.153418 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:09:12.154373 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.156025 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.158634 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.159644 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.159770 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:12.160028 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:12.161080 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0926 03:09:12.161287 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":390,"threads_started":1}
I20260926 03:09:12.162869 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:12.149689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43667 (local address 127.0.0.1:42328)
0926 03:09:12.151837 (+ 2148us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:12.151873 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:12.152027 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:12.154880 (+ 2853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:12.154886 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:12.154917 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:09:12.155268 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:12.155279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.157345 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.157350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:12.159704 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:12.159720 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.161674 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.161681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:12.161856 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:12.161895 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 03:09:12.162065 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 03:09:12.162402 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1899,"mutex_wait_us":48,"thread_start_us":2592,"threads_started":1}
I20260926 03:09:12.162974 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:12.149689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44205 (local address 127.0.0.1:41288)
0926 03:09:12.152565 (+ 2876us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:12.152583 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:12.152701 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:12.153685 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:12.153694 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:12.153727 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:09:12.154165 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:12.154179 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.156164 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.156169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:12.158435 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:12.158447 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.159898 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.159905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:12.160429 (+ 524us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:12.160541 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 03:09:12.162220 (+ 1679us) client_negotiation.cc:240] Negotiation successful
0926 03:09:12.162551 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2736,"mutex_wait_us":331,"thread_start_us":1568,"threads_started":1}
I20260926 03:09:12.163340 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:12.150168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42328 (local address 127.0.97.1:43667)
0926 03:09:12.154194 (+ 4026us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:12.154205 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:12.154235 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:09:12.154334 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:12.154342 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:12.154656 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:12.155512 (+ 856us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:09:12.155744 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.157178 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.159923 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.161335 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.162393 (+ 1058us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:12.162776 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:12.162859 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 03:09:12.163041 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3826,"thread_start_us":3653,"threads_started":1}
I20260926 03:09:12.164191 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:12.152752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37055 (local address 127.0.0.1:52860)
0926 03:09:12.154932 (+ 2180us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:12.154949 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:12.155105 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:12.155653 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:12.155657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:12.155680 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:09:12.156007 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:12.156016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.157861 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.157866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:12.160425 (+ 2559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:12.160442 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.163367 (+ 2925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.163374 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:12.163562 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:12.163596 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 03:09:12.163692 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 03:09:12.163829 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2057,"mutex_wait_us":123,"spinlock_wait_cycles":4352,"thread_start_us":1920,"threads_started":1}
I20260926 03:09:12.164263 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:12.153282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52860 (local address 127.0.97.2:37055)
0926 03:09:12.154472 (+ 1190us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:12.154481 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:12.155068 (+ 587us) server_negotiation.cc:407] Connection header received
0926 03:09:12.155262 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:12.155267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:12.155459 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:12.155629 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:09:12.156174 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.157697 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.160805 (+ 3108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.162568 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.162717 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:12.162937 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:12.163767 (+ 830us) server_negotiation.cc:299] Negotiation successful
0926 03:09:12.163947 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1021,"thread_start_us":899,"threads_started":1}
I20260926 03:09:12.167469 13730 tablet_service.cc:1467] Processing CreateTablet for tablet 51fd8c9eabac4c4e828bc27f5e29637a (DEFAULT_TABLE table=test-table [id=5dcef5bdaffc40e9afbae36529434c8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:09:12.167469 13451 tablet_service.cc:1467] Processing CreateTablet for tablet 51fd8c9eabac4c4e828bc27f5e29637a (DEFAULT_TABLE table=test-table [id=5dcef5bdaffc40e9afbae36529434c8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:09:12.169159 13451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51fd8c9eabac4c4e828bc27f5e29637a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:09:12.169142 13730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51fd8c9eabac4c4e828bc27f5e29637a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:09:12.170258 13596 tablet_service.cc:1467] Processing CreateTablet for tablet 51fd8c9eabac4c4e828bc27f5e29637a (DEFAULT_TABLE table=test-table [id=5dcef5bdaffc40e9afbae36529434c8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:09:12.171571 13596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51fd8c9eabac4c4e828bc27f5e29637a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:09:12.181118 13661 heartbeater.cc:503] Master 127.0.97.62:40575 was elected leader, sending a full tablet report...
I20260926 03:09:12.185320 13823 tablet_bootstrap.cc:492] T 51fd8c9eabac4c4e828bc27f5e29637a P bf646bb91aac46c5890621f5cf8b6d5b: Bootstrap starting.
I20260926 03:09:12.187630 13824 tablet_bootstrap.cc:492] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579: Bootstrap starting.
I20260926 03:09:12.189947 13823 tablet_bootstrap.cc:654] T 51fd8c9eabac4c4e828bc27f5e29637a P bf646bb91aac46c5890621f5cf8b6d5b: Neither blocks nor log segments found. Creating new log.
I20260926 03:09:12.191643 13824 tablet_bootstrap.cc:654] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579: Neither blocks nor log segments found. Creating new log.
I20260926 03:09:12.191998 13823 log.cc:824] T 51fd8c9eabac4c4e828bc27f5e29637a P bf646bb91aac46c5890621f5cf8b6d5b: Log is configured to *not* fsync() on all Append() calls
I20260926 03:09:12.193014 13824 log.cc:824] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579: Log is configured to *not* fsync() on all Append() calls
I20260926 03:09:12.194267 13825 tablet_bootstrap.cc:492] T 51fd8c9eabac4c4e828bc27f5e29637a P 6d4d384eecb34663ab6522e0392e065f: Bootstrap starting.
I20260926 03:09:12.195683 13823 tablet_bootstrap.cc:492] T 51fd8c9eabac4c4e828bc27f5e29637a P bf646bb91aac46c5890621f5cf8b6d5b: No bootstrap required, opened a new log
I20260926 03:09:12.196015 13823 ts_tablet_manager.cc:1397] T 51fd8c9eabac4c4e828bc27f5e29637a P bf646bb91aac46c5890621f5cf8b6d5b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260926 03:09:12.196185 13824 tablet_bootstrap.cc:492] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579: No bootstrap required, opened a new log
I20260926 03:09:12.196436 13824 ts_tablet_manager.cc:1397] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 03:09:12.198119 13825 tablet_bootstrap.cc:654] T 51fd8c9eabac4c4e828bc27f5e29637a P 6d4d384eecb34663ab6522e0392e065f: Neither blocks nor log segments found. Creating new log.
I20260926 03:09:12.199544 13825 log.cc:824] T 51fd8c9eabac4c4e828bc27f5e29637a P 6d4d384eecb34663ab6522e0392e065f: Log is configured to *not* fsync() on all Append() calls
I20260926 03:09:12.204312 13825 tablet_bootstrap.cc:492] T 51fd8c9eabac4c4e828bc27f5e29637a P 6d4d384eecb34663ab6522e0392e065f: No bootstrap required, opened a new log
I20260926 03:09:12.204654 13825 ts_tablet_manager.cc:1397] T 51fd8c9eabac4c4e828bc27f5e29637a P 6d4d384eecb34663ab6522e0392e065f: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260926 03:09:12.204639 13824 raft_consensus.cc:348] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4d384eecb34663ab6522e0392e065f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44205 } } peers { permanent_uuid: "bf646bb91aac46c5890621f5cf8b6d5b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43667 } } peers { permanent_uuid: "44fdbdfab23f44d09972b372222da579" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } }
I20260926 03:09:12.205044 13824 raft_consensus.cc:374] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:09:12.205175 13824 raft_consensus.cc:729] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44fdbdfab23f44d09972b372222da579, State: Initialized, Role: FOLLOWER
I20260926 03:09:12.205812 13824 consensus_queue.cc:260] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4d384eecb34663ab6522e0392e065f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44205 } } peers { permanent_uuid: "bf646bb91aac46c5890621f5cf8b6d5b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43667 } } peers { permanent_uuid: "44fdbdfab23f44d09972b372222da579" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } }
I20260926 03:09:12.207721 13824 ts_tablet_manager.cc:1428] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579: Time spent starting tablet: real 0.011s user 0.006s sys 0.006s
I20260926 03:09:12.208545 13823 raft_consensus.cc:348] T 51fd8c9eabac4c4e828bc27f5e29637a P bf646bb91aac46c5890621f5cf8b6d5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4d384eecb34663ab6522e0392e065f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44205 } } peers { permanent_uuid: "bf646bb91aac46c5890621f5cf8b6d5b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43667 } } peers { permanent_uuid: "44fdbdfab23f44d09972b372222da579" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } }
I20260926 03:09:12.210647 13823 raft_consensus.cc:374] T 51fd8c9eabac4c4e828bc27f5e29637a P bf646bb91aac46c5890621f5cf8b6d5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:09:12.210814 13823 raft_consensus.cc:729] T 51fd8c9eabac4c4e828bc27f5e29637a P bf646bb91aac46c5890621f5cf8b6d5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf646bb91aac46c5890621f5cf8b6d5b, State: Initialized, Role: FOLLOWER
I20260926 03:09:12.211508 13823 consensus_queue.cc:260] T 51fd8c9eabac4c4e828bc27f5e29637a P bf646bb91aac46c5890621f5cf8b6d5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4d384eecb34663ab6522e0392e065f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44205 } } peers { permanent_uuid: "bf646bb91aac46c5890621f5cf8b6d5b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43667 } } peers { permanent_uuid: "44fdbdfab23f44d09972b372222da579" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } }
I20260926 03:09:12.213129 13516 heartbeater.cc:503] Master 127.0.97.62:40575 was elected leader, sending a full tablet report...
I20260926 03:09:12.213564 13823 ts_tablet_manager.cc:1428] T 51fd8c9eabac4c4e828bc27f5e29637a P bf646bb91aac46c5890621f5cf8b6d5b: Time spent starting tablet: real 0.017s user 0.011s sys 0.003s
I20260926 03:09:12.214885 13825 raft_consensus.cc:348] T 51fd8c9eabac4c4e828bc27f5e29637a P 6d4d384eecb34663ab6522e0392e065f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4d384eecb34663ab6522e0392e065f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44205 } } peers { permanent_uuid: "bf646bb91aac46c5890621f5cf8b6d5b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43667 } } peers { permanent_uuid: "44fdbdfab23f44d09972b372222da579" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } }
I20260926 03:09:12.215435 13825 raft_consensus.cc:374] T 51fd8c9eabac4c4e828bc27f5e29637a P 6d4d384eecb34663ab6522e0392e065f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:09:12.215696 13825 raft_consensus.cc:729] T 51fd8c9eabac4c4e828bc27f5e29637a P 6d4d384eecb34663ab6522e0392e065f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4d384eecb34663ab6522e0392e065f, State: Initialized, Role: FOLLOWER
I20260926 03:09:12.216456 13825 consensus_queue.cc:260] T 51fd8c9eabac4c4e828bc27f5e29637a P 6d4d384eecb34663ab6522e0392e065f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4d384eecb34663ab6522e0392e065f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44205 } } peers { permanent_uuid: "bf646bb91aac46c5890621f5cf8b6d5b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43667 } } peers { permanent_uuid: "44fdbdfab23f44d09972b372222da579" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } }
I20260926 03:09:12.217702 13797 heartbeater.cc:503] Master 127.0.97.62:40575 was elected leader, sending a full tablet report...
I20260926 03:09:12.218422 13825 ts_tablet_manager.cc:1428] T 51fd8c9eabac4c4e828bc27f5e29637a P 6d4d384eecb34663ab6522e0392e065f: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260926 03:09:12.332031 13805 tablet.cc:2369] T 51fd8c9eabac4c4e828bc27f5e29637a P 6d4d384eecb34663ab6522e0392e065f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:09:12.350934 13829 raft_consensus.cc:482] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:09:12.351279 13829 raft_consensus.cc:504] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4d384eecb34663ab6522e0392e065f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44205 } } peers { permanent_uuid: "bf646bb91aac46c5890621f5cf8b6d5b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43667 } } peers { permanent_uuid: "44fdbdfab23f44d09972b372222da579" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } }
I20260926 03:09:12.353080 13829 leader_election.cc:288] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4d384eecb34663ab6522e0392e065f (127.0.97.3:44205), bf646bb91aac46c5890621f5cf8b6d5b (127.0.97.1:43667)
I20260926 03:09:12.360224 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:12.353568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43771 (local address 127.0.97.3:44205)
0926 03:09:12.353777 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:12.353784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:12.354567 (+ 783us) server_negotiation.cc:407] Connection header received
0926 03:09:12.354656 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:12.354662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:12.354736 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:12.354823 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:09:12.355581 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.356363 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.358341 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.359198 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.359305 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:12.359391 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:12.359831 (+ 440us) server_negotiation.cc:299] Negotiation successful
0926 03:09:12.359971 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 03:09:12.360278 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:12.353637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52041 (local address 127.0.97.1:43667)
0926 03:09:12.353836 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:12.353841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:12.354572 (+ 731us) server_negotiation.cc:407] Connection header received
0926 03:09:12.354917 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:12.354924 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:12.355025 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:12.355133 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:09:12.355813 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.356611 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.358523 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.359370 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.359498 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:12.359601 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:12.359908 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 03:09:12.360055 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:09:12.360347 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:12.353606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43667 (local address 127.0.97.2:52041)
0926 03:09:12.354441 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:12.354464 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:12.354680 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:12.355188 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:12.355195 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:12.355217 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:09:12.355626 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:12.355639 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.356788 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.356792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:12.358325 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:12.358339 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.359600 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.359605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:12.359721 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:12.359749 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 03:09:12.359841 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 03:09:12.359998 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":419,"threads_started":1}
I20260926 03:09:12.360369 13832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:12.353513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44205 (local address 127.0.97.2:43771)
0926 03:09:12.354355 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:12.354371 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:12.354542 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:12.354908 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:12.354926 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:12.354955 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:09:12.355333 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:12.355344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.356540 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.356545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:12.358124 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:12.358138 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.359441 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.359448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:12.359650 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:12.359677 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 03:09:12.359756 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 03:09:12.359908 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":5,"thread_start_us":492,"threads_started":1}
I20260926 03:09:12.361493 13471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fd8c9eabac4c4e828bc27f5e29637a" candidate_uuid: "44fdbdfab23f44d09972b372222da579" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf646bb91aac46c5890621f5cf8b6d5b" is_pre_election: true
I20260926 03:09:12.361866 13750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fd8c9eabac4c4e828bc27f5e29637a" candidate_uuid: "44fdbdfab23f44d09972b372222da579" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4d384eecb34663ab6522e0392e065f" is_pre_election: true
I20260926 03:09:12.362159 13471 raft_consensus.cc:2454] T 51fd8c9eabac4c4e828bc27f5e29637a P bf646bb91aac46c5890621f5cf8b6d5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44fdbdfab23f44d09972b372222da579 in term 0.
I20260926 03:09:12.362385 13750 raft_consensus.cc:2454] T 51fd8c9eabac4c4e828bc27f5e29637a P 6d4d384eecb34663ab6522e0392e065f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44fdbdfab23f44d09972b372222da579 in term 0.
I20260926 03:09:12.362938 13550 leader_election.cc:302] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44fdbdfab23f44d09972b372222da579, bf646bb91aac46c5890621f5cf8b6d5b; no voters:
I20260926 03:09:12.363317 13829 raft_consensus.cc:2781] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:09:12.363452 13829 raft_consensus.cc:482] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:09:12.363554 13829 raft_consensus.cc:3037] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:09:12.365914 13829 raft_consensus.cc:504] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4d384eecb34663ab6522e0392e065f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44205 } } peers { permanent_uuid: "bf646bb91aac46c5890621f5cf8b6d5b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43667 } } peers { permanent_uuid: "44fdbdfab23f44d09972b372222da579" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } }
I20260926 03:09:12.366595 13829 leader_election.cc:288] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579 [CANDIDATE]: Term 1 election: Requested vote from peers 6d4d384eecb34663ab6522e0392e065f (127.0.97.3:44205), bf646bb91aac46c5890621f5cf8b6d5b (127.0.97.1:43667)
I20260926 03:09:12.367182 13471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fd8c9eabac4c4e828bc27f5e29637a" candidate_uuid: "44fdbdfab23f44d09972b372222da579" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf646bb91aac46c5890621f5cf8b6d5b"
I20260926 03:09:12.367190 13750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51fd8c9eabac4c4e828bc27f5e29637a" candidate_uuid: "44fdbdfab23f44d09972b372222da579" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4d384eecb34663ab6522e0392e065f"
I20260926 03:09:12.367405 13471 raft_consensus.cc:3037] T 51fd8c9eabac4c4e828bc27f5e29637a P bf646bb91aac46c5890621f5cf8b6d5b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:09:12.367396 13750 raft_consensus.cc:3037] T 51fd8c9eabac4c4e828bc27f5e29637a P 6d4d384eecb34663ab6522e0392e065f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:09:12.369853 13750 raft_consensus.cc:2454] T 51fd8c9eabac4c4e828bc27f5e29637a P 6d4d384eecb34663ab6522e0392e065f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44fdbdfab23f44d09972b372222da579 in term 1.
I20260926 03:09:12.370371 13549 leader_election.cc:302] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44fdbdfab23f44d09972b372222da579, 6d4d384eecb34663ab6522e0392e065f; no voters:
I20260926 03:09:12.370848 13829 raft_consensus.cc:2781] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:09:12.371069 13471 raft_consensus.cc:2454] T 51fd8c9eabac4c4e828bc27f5e29637a P bf646bb91aac46c5890621f5cf8b6d5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44fdbdfab23f44d09972b372222da579 in term 1.
I20260926 03:09:12.371526 13829 raft_consensus.cc:686] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579 [term 1 LEADER]: Becoming Leader. State: Replica: 44fdbdfab23f44d09972b372222da579, State: Running, Role: LEADER
I20260926 03:09:12.372100 13829 consensus_queue.cc:237] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4d384eecb34663ab6522e0392e065f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44205 } } peers { permanent_uuid: "bf646bb91aac46c5890621f5cf8b6d5b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43667 } } peers { permanent_uuid: "44fdbdfab23f44d09972b372222da579" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } }
I20260926 03:09:12.377158 13317 catalog_manager.cc:5495] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44fdbdfab23f44d09972b372222da579 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "44fdbdfab23f44d09972b372222da579" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4d384eecb34663ab6522e0392e065f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf646bb91aac46c5890621f5cf8b6d5b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44fdbdfab23f44d09972b372222da579" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } health_report { overall_health: HEALTHY } } }
W20260926 03:09:12.386934 13662 tablet.cc:2369] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:09:12.404703 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:12.398831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52870 (local address 127.0.97.2:37055)
0926 03:09:12.399045 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:12.399050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:12.399115 (+ 65us) server_negotiation.cc:407] Connection header received
0926 03:09:12.399320 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:12.399323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:12.399417 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:12.399607 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:09:12.400024 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.400812 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.402243 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.402763 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.402869 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:12.403560 (+ 691us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:09:12.404116 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:09:12.404259 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:12.404373 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 03:09:12.404448 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 03:09:12.412698 13517 tablet.cc:2369] T 51fd8c9eabac4c4e828bc27f5e29637a P bf646bb91aac46c5890621f5cf8b6d5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:09:12.414716 13750 raft_consensus.cc:1261] T 51fd8c9eabac4c4e828bc27f5e29637a P 6d4d384eecb34663ab6522e0392e065f [term 1 FOLLOWER]: Refusing update from remote peer 44fdbdfab23f44d09972b372222da579: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:09:12.414742 13471 raft_consensus.cc:1261] T 51fd8c9eabac4c4e828bc27f5e29637a P bf646bb91aac46c5890621f5cf8b6d5b [term 1 FOLLOWER]: Refusing update from remote peer 44fdbdfab23f44d09972b372222da579: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:09:12.415869 13829 consensus_queue.cc:1035] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf646bb91aac46c5890621f5cf8b6d5b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:09:12.416200 13834 consensus_queue.cc:1035] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4d384eecb34663ab6522e0392e065f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:09:12.426987 13837 mvcc.cc:205] Tried to move back new op lower bound from 7333446256278392832 to 7333446256120184832. Current Snapshot: MvccSnapshot[applied={T|T < 7333446256278392832}]
I20260926 03:09:12.433837 13839 mvcc.cc:205] Tried to move back new op lower bound from 7333446256278392832 to 7333446256120184832. Current Snapshot: MvccSnapshot[applied={T|T < 7333446256278392832}]
I20260926 03:09:12.445858 13318 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:09:12.446225 13318 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:09:12.450560 13318 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cfdf4b81dcd645609d5d20f56d6497a1: Sending DeleteTablet for 3 replicas of tablet 51fd8c9eabac4c4e828bc27f5e29637a
I20260926 03:09:12.451967 13451 tablet_service.cc:1514] Processing DeleteTablet for tablet 51fd8c9eabac4c4e828bc27f5e29637a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:09:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42328
I20260926 03:09:12.451958 13730 tablet_service.cc:1514] Processing DeleteTablet for tablet 51fd8c9eabac4c4e828bc27f5e29637a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:09:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41288
I20260926 03:09:12.457083 13596 tablet_service.cc:1514] Processing DeleteTablet for tablet 51fd8c9eabac4c4e828bc27f5e29637a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:09:12 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52860
I20260926 03:09:12.459852 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 13355
I20260926 03:09:12.467097 13848 tablet_replica.cc:330] T 51fd8c9eabac4c4e828bc27f5e29637a P 6d4d384eecb34663ab6522e0392e065f: stopping tablet replica
I20260926 03:09:12.468853 13847 tablet_replica.cc:330] T 51fd8c9eabac4c4e828bc27f5e29637a P bf646bb91aac46c5890621f5cf8b6d5b: stopping tablet replica
I20260926 03:09:12.469165 13848 raft_consensus.cc:2229] T 51fd8c9eabac4c4e828bc27f5e29637a P 6d4d384eecb34663ab6522e0392e065f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:09:12.469347 13847 raft_consensus.cc:2229] T 51fd8c9eabac4c4e828bc27f5e29637a P bf646bb91aac46c5890621f5cf8b6d5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:09:12.469753 13848 raft_consensus.cc:2258] T 51fd8c9eabac4c4e828bc27f5e29637a P 6d4d384eecb34663ab6522e0392e065f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:09:12.469812 13847 raft_consensus.cc:2258] T 51fd8c9eabac4c4e828bc27f5e29637a P bf646bb91aac46c5890621f5cf8b6d5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:09:12.472199 13848 ts_tablet_manager.cc:1905] T 51fd8c9eabac4c4e828bc27f5e29637a P 6d4d384eecb34663ab6522e0392e065f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:09:12.478950 13848 ts_tablet_manager.cc:1918] T 51fd8c9eabac4c4e828bc27f5e29637a P 6d4d384eecb34663ab6522e0392e065f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:09:12.479144 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:12.461088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42338 (local address 127.0.97.1:43667)
0926 03:09:12.461288 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:12.461295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:12.461325 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:09:12.461530 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:12.461536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:12.461625 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:12.461749 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:12.464066 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.465330 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.467203 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.467954 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.468094 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:12.471582 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:12.471600 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:12.471619 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:12.471667 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:12.475355 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:12.476414 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:12.476425 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:12.476428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:12.476620 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:12.477295 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:12.477301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:12.477304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:12.477787 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:12.478110 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:12.478410 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 03:09:12.478762 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:09:12.479264 13848 log.cc:1196] T 51fd8c9eabac4c4e828bc27f5e29637a P 6d4d384eecb34663ab6522e0392e065f: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-2/wal/wals/51fd8c9eabac4c4e828bc27f5e29637a
I20260926 03:09:12.479875 13848 ts_tablet_manager.cc:1939] T 51fd8c9eabac4c4e828bc27f5e29637a P 6d4d384eecb34663ab6522e0392e065f: Deleting consensus metadata
I20260926 03:09:12.480093 13511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:09:12.482230 13303 catalog_manager.cc:4841] TS 6d4d384eecb34663ab6522e0392e065f (127.0.97.3:44205): tablet 51fd8c9eabac4c4e828bc27f5e29637a (table test-table [id=5dcef5bdaffc40e9afbae36529434c8b]) successfully deleted
I20260926 03:09:12.482996 13849 tablet_replica.cc:330] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579: stopping tablet replica
I20260926 03:09:12.483650 13849 raft_consensus.cc:2229] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:09:12.485060 13849 raft_consensus.cc:2258] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:09:12.487399 13849 ts_tablet_manager.cc:1905] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:09:12.495702 13849 ts_tablet_manager.cc:1918] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:09:12.495939 13849 log.cc:1196] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391950058021-388-0/minicluster-data/ts-1/wal/wals/51fd8c9eabac4c4e828bc27f5e29637a
I20260926 03:09:12.496554 13849 ts_tablet_manager.cc:1939] T 51fd8c9eabac4c4e828bc27f5e29637a P 44fdbdfab23f44d09972b372222da579: Deleting consensus metadata
I20260926 03:09:12.498708 13304 catalog_manager.cc:4841] TS 44fdbdfab23f44d09972b372222da579 (127.0.97.2:37055): tablet 51fd8c9eabac4c4e828bc27f5e29637a (table test-table [id=5dcef5bdaffc40e9afbae36529434c8b]) successfully deleted
I20260926 03:09:12.483222 13847 ts_tablet_manager.cc:1905] T 51fd8c9eabac4c4e828bc27f5e29637a P bf646bb91aac46c5890621f5cf8b6d5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:09:12.585278 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 13355
W20260926 03:09:12.599951 13304 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:43667 (error 108)
I20260926 03:09:12.600672 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 13520
I20260926 03:09:12.601678 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:12.601185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43667 (local address 127.0.0.1:42346)
0926 03:09:12.601441 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:12.601541 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260926 03:09:12.601928 13304 catalog_manager.cc:4584] TS bf646bb91aac46c5890621f5cf8b6d5b (127.0.97.1:43667): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 51fd8c9eabac4c4e828bc27f5e29637a: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43667: connect: Connection refused (error 111)
I20260926 03:09:12.616811 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:12.601363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52876 (local address 127.0.97.2:37055)
0926 03:09:12.601591 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:12.601598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:12.601993 (+ 395us) server_negotiation.cc:407] Connection header received
0926 03:09:12.602061 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:12.602066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:12.602156 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:12.602250 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:12.603624 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.604894 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.606271 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.607119 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.607254 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:12.609841 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:12.609859 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:12.609880 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:12.609937 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:12.613803 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:12.614620 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:12.614630 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:12.614634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:12.614838 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:12.615191 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:12.615195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:12.615199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:12.615668 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:12.615975 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:12.616185 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 03:09:12.616427 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 03:09:12.617782 13656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:09:12.655862 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:12.655301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43667 (local address 127.0.0.1:42352)
0926 03:09:12.655549 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:12.655674 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 03:09:12.656961 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:12.656585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43667 (local address 127.0.0.1:42360)
0926 03:09:12.656730 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:12.656828 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 03:09:12.717484 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:12.716942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43667 (local address 127.0.0.1:42364)
0926 03:09:12.717186 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:12.717300 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 03:09:12.718542 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:12.718139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43667 (local address 127.0.0.1:42372)
0926 03:09:12.718282 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:12.718377 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 03:09:12.734063 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 13520
I20260926 03:09:12.749254 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 13665
I20260926 03:09:12.762907 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:12.750015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41294 (local address 127.0.97.3:44205)
0926 03:09:12.750210 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:12.750216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:12.750239 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:09:12.750413 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:12.750419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:12.750528 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:12.750627 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:12.751913 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.752639 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.753995 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:12.754457 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:12.754627 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:12.757321 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:12.757336 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:12.757352 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:12.757405 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:12.760086 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:12.760805 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:12.760813 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:12.760815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:12.760960 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:12.761443 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:12.761447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:12.761450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:12.761829 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:12.762073 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:12.762331 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 03:09:12.762544 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:09:12.763675 13790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:09:12.799693 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:12.799174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43667 (local address 127.0.0.1:42374)
0926 03:09:12.799391 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:12.799492 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 03:09:12.800989 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:12.800524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43667 (local address 127.0.0.1:42384)
0926 03:09:12.800738 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:12.800848 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260926 03:09:12.855921 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 13665
I20260926 03:09:12.869454 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 13285
I20260926 03:09:12.870769 13348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:09:12.982283 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:12.981619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43667 (local address 127.0.0.1:42396)
0926 03:09:12.981845 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:12.981991 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 03:09:12.982578 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 13285
I20260926 03:09:13.000370 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 03:09:13.336705 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 03:09:13.336925 388 mini_ranger.cc:66] Stopping Ranger...
I20260926 03:09:13.674785 388 mini_ranger.cc:68] Stopped Ranger
2026-09-26 03:09:13.675 UTC [12336] LOG: received smart shutdown request
2026-09-26 03:09:13.679 UTC [12336] LOG: background worker "logical replication launcher" (PID 12345) exited with exit code 1
2026-09-26 03:09:13.679 UTC [12340] LOG: shutting down
2026-09-26 03:09:13.922 UTC [12336] LOG: database system is shut down
2026-09-26T03:09:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55734 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:09:14 dist-test-slave-hmbw krb5kdc[13863](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:09:14 dist-test-slave-hmbw krb5kdc[13863](info): set up 2 sockets
Sep 26 03:09:14 dist-test-slave-hmbw krb5kdc[13863](info): commencing operation
krb5kdc: starting...
W20260926 03:09:15.059309 388 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:09:15 dist-test-slave-hmbw krb5kdc[13863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:09:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:09:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:09:15.291754 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33607
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33607,127.0.97.61:38657,127.0.97.60:38083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:09:15.624854 13876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:09:15.625164 13876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:09:15.625283 13876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:09:15.633795 13876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:09:15.633919 13876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:09:15.633980 13876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:09:15.634064 13876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:09:15.634124 13876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:09:15.644564 13876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33607,127.0.97.61:38657,127.0.97.60:38083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:09:15.645886 13876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:09:15.647265 13876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:09:15.659986 13881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:15.660072 13882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:15.660353 13876 server_base.cc:1034] running on GCE node
W20260926 03:09:15.660045 13884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:15.661509 13876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:09:15.663123 13876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:09:15.664332 13876 hybrid_clock.cc:648] HybridClock initialized: now 1790392155664277 us; error 58 us; skew 500 ppm
I20260926 03:09:15.664690 13876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:09:15 dist-test-slave-hmbw krb5kdc[13863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:09:15.684019 13876 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:09:15.685775 13876 webserver.cc:460] Webserver started at http://127.0.97.62:36347/ using document root <none> and password file <none>
I20260926 03:09:15.686589 13876 fs_manager.cc:359] Metadata directory not provided
I20260926 03:09:15.686723 13876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:09:15.687038 13876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:09:15.690313 13876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "5a758ee4c4834b63be4a6ff739daa3b1"
format_stamp: "Formatted at 2026-09-26 03:09:15 on dist-test-slave-hmbw"
server_key: "c41a321cd9df39691e4f85c4bf93594b"
server_key_iv: "c4bd69971c68de3240de72615247cd22"
server_key_version: "encryptionkey@0"
I20260926 03:09:15.691234 13876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "5a758ee4c4834b63be4a6ff739daa3b1"
format_stamp: "Formatted at 2026-09-26 03:09:15 on dist-test-slave-hmbw"
server_key: "c41a321cd9df39691e4f85c4bf93594b"
server_key_iv: "c4bd69971c68de3240de72615247cd22"
server_key_version: "encryptionkey@0"
I20260926 03:09:15.696978 13876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:09:15.700953 13891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:15.702407 13876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:09:15.702643 13876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "5a758ee4c4834b63be4a6ff739daa3b1"
format_stamp: "Formatted at 2026-09-26 03:09:15 on dist-test-slave-hmbw"
server_key: "c41a321cd9df39691e4f85c4bf93594b"
server_key_iv: "c4bd69971c68de3240de72615247cd22"
server_key_version: "encryptionkey@0"
I20260926 03:09:15.702853 13876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:09:15.722460 13876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:09:15.726960 13876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:09:15.727236 13876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:09:15.745942 13876 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33607
I20260926 03:09:15.745994 13942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33607 every 8 connection(s)
I20260926 03:09:15.747364 13876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:09:15.751511 13943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:09:15.754807 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 13876
I20260926 03:09:15.755010 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:09:15.755725 388 external_mini_cluster.cc:1428] Setting key ee301836f3f513433465afee95b97361
I20260926 03:09:15.758620 13943 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33607 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:09:15 dist-test-slave-hmbw krb5kdc[13863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260926 03:09:15.783004 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:15.761918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60986 (local address 127.0.97.62:33607)
0926 03:09:15.763036 (+ 1118us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:15.763043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:15.763166 (+ 123us) server_negotiation.cc:407] Connection header received
0926 03:09:15.765410 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:15.765419 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:15.765759 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:15.765984 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:15.767387 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:15.769204 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:15.770734 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:15.771653 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:15.771854 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:15.775428 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:15.775439 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:15.775453 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:15.775491 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:15.778452 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:15.779439 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:15.779448 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:15.779450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:15.779621 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:15.780062 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:15.780065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:15.780067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:15.780682 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:15.780832 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:15.781559 (+ 727us) server_negotiation.cc:299] Negotiation successful
0926 03:09:15.781849 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":905,"spinlock_wait_cycles":32128,"thread_start_us":686,"threads_started":1}
I20260926 03:09:15.783044 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:15.761917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33607 (local address 127.0.0.1:60986)
0926 03:09:15.762946 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:15.763012 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:15.764948 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:15.766030 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:15.766036 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:15.766760 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:09:15.767143 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:15.767163 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:15.769402 (+ 2239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:15.769406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:15.770586 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:15.770601 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:15.771905 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:15.771911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:15.772039 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:15.772720 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:09:15.772749 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:09:15.775113 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:09:15.778631 (+ 3518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:15.778644 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:15.778658 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:15.779212 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:15.779747 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:15.779751 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:15.779755 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:15.779960 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:15.780825 (+ 865us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:09:15.780834 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:09:15.781110 (+ 276us) client_negotiation.cc:769] Sending connection context
0926 03:09:15.781372 (+ 262us) client_negotiation.cc:240] Negotiation successful
0926 03:09:15.782015 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":104,"thread_start_us":494,"threads_started":1}
I20260926 03:09:15.785952 13943 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38657 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:09:15.787081 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:15.786686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38657 (local address 127.0.0.1:52718)
0926 03:09:15.786854 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:15.786960 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 03:09:15.787344 13893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38657: connect: Connection refused (error 111)
I20260926 03:09:15.788975 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:15.788558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38657 (local address 127.0.0.1:52732)
0926 03:09:15.788738 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:15.788816 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260926 03:09:15.789201 13943 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38657: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38657: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260926 03:09:15.807475 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38657
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33607,127.0.97.61:38657,127.0.97.60:38083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:09:15.838642 13943 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38657 } attempt: 1
I20260926 03:09:15.839851 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:15.839398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38657 (local address 127.0.0.1:52748)
0926 03:09:15.839607 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:15.839716 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 03:09:15.840982 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:15.840596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38657 (local address 127.0.0.1:52756)
0926 03:09:15.840760 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:15.840834 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 03:09:15.841182 13943 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38657: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38657: connect: Connection refused (error 111)
I20260926 03:09:15.909444 13943 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38657 } attempt: 2
I20260926 03:09:15.910866 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:15.910225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38657 (local address 127.0.0.1:52758)
0926 03:09:15.910620 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:15.910715 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260926 03:09:15.912118 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:15.911659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38657 (local address 127.0.0.1:52764)
0926 03:09:15.911855 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:15.911943 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260926 03:09:15.912333 13943 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38657: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38657: connect: Connection refused (error 111)
I20260926 03:09:16.003618 13943 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38657 } attempt: 3
I20260926 03:09:16.004874 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.004374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38657 (local address 127.0.0.1:52778)
0926 03:09:16.004629 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.004725 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 03:09:16.006199 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.005703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38657 (local address 127.0.0.1:52794)
0926 03:09:16.005915 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.006007 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260926 03:09:16.006378 13943 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38657: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38657: connect: Connection refused (error 111)
W20260926 03:09:16.142333 13950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:09:16.142714 13950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:09:16.142856 13950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 03:09:16.149747 13943 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38657 } attempt: 4
W20260926 03:09:16.150609 13950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:09:16.150738 13950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:09:16.150805 13950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:09:16.150878 13950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:09:16.150940 13950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:09:16.151213 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.150538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38657 (local address 127.0.0.1:52806)
0926 03:09:16.150949 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.151071 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209}
I20260926 03:09:16.152313 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.151920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38657 (local address 127.0.0.1:52812)
0926 03:09:16.152087 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.152186 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 03:09:16.152506 13943 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38657: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38657: connect: Connection refused (error 111)
I20260926 03:09:16.161759 13950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33607,127.0.97.61:38657,127.0.97.60:38083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:09:16.163223 13950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:09:16.164561 13950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:09:16.177030 13955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:16.176995 13956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:16.176957 13958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:16.178004 13950 server_base.cc:1034] running on GCE node
I20260926 03:09:16.178901 13950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:09:16.180490 13950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:09:16.181763 13950 hybrid_clock.cc:648] HybridClock initialized: now 1790392156181708 us; error 43 us; skew 500 ppm
I20260926 03:09:16.182157 13950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:09:16 dist-test-slave-hmbw krb5kdc[13863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:09:16.185794 13950 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260926 03:09:16.187577 13950 webserver.cc:460] Webserver started at http://127.0.97.61:34983/ using document root <none> and password file <none>
I20260926 03:09:16.188333 13950 fs_manager.cc:359] Metadata directory not provided
I20260926 03:09:16.188486 13950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:09:16.188802 13950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:09:16.192233 13950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-1/data/instance:
uuid: "f017d75d274140819f9092d957859922"
format_stamp: "Formatted at 2026-09-26 03:09:16 on dist-test-slave-hmbw"
server_key: "03c97aec69d14b87acec9f3228bcdc93"
server_key_iv: "1430932fc9c30b21cedc9145a2431c21"
server_key_version: "encryptionkey@0"
I20260926 03:09:16.193110 13950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-1/wal/instance:
uuid: "f017d75d274140819f9092d957859922"
format_stamp: "Formatted at 2026-09-26 03:09:16 on dist-test-slave-hmbw"
server_key: "03c97aec69d14b87acec9f3228bcdc93"
server_key_iv: "1430932fc9c30b21cedc9145a2431c21"
server_key_version: "encryptionkey@0"
I20260926 03:09:16.199779 13950 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 03:09:16.203915 13965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:16.205596 13950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:09:16.205827 13950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-1/wal
uuid: "f017d75d274140819f9092d957859922"
format_stamp: "Formatted at 2026-09-26 03:09:16 on dist-test-slave-hmbw"
server_key: "03c97aec69d14b87acec9f3228bcdc93"
server_key_iv: "1430932fc9c30b21cedc9145a2431c21"
server_key_version: "encryptionkey@0"
I20260926 03:09:16.206054 13950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:09:16.227993 13950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:09:16.233834 13950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:09:16.234193 13950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:09:16.253935 13950 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:38657
I20260926 03:09:16.253969 14016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:38657 every 8 connection(s)
I20260926 03:09:16.255417 13950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-1/data/info.pb
I20260926 03:09:16.259456 14017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:09:16.262920 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 13950
I20260926 03:09:16.263129 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-1/wal/instance
I20260926 03:09:16.263648 388 external_mini_cluster.cc:1428] Setting key 29e350c643fb61ad86c6b5180296f6b9
I20260926 03:09:16.268080 14017 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33607 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:09:16 dist-test-slave-hmbw krb5kdc[13863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260926 03:09:16.292088 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.271699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60988 (local address 127.0.97.62:33607)
0926 03:09:16.271930 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:16.271935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:16.273320 (+ 1385us) server_negotiation.cc:407] Connection header received
0926 03:09:16.275530 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:16.275537 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:16.275630 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:16.275758 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:16.277443 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.278146 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.279919 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.280446 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.280577 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:16.284520 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:16.284528 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:16.284533 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:16.284592 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:16.288198 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:16.289188 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:16.289194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:16.289198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:16.289368 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:16.289870 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:16.289875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:16.289878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:16.290254 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:16.290430 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:16.291530 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0926 03:09:16.291778 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 03:09:16.293043 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.272226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33607 (local address 127.0.0.1:60988)
0926 03:09:16.273131 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.273171 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:16.275167 (+ 1996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:16.276023 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:16.276033 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:16.276606 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:09:16.277226 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:16.277246 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.278335 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.278340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:16.279732 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:16.279748 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.280720 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.280729 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:16.280957 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:16.281639 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:09:16.281663 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:09:16.284202 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:09:16.288456 (+ 4254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:16.288467 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:16.288484 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:16.289021 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:16.289516 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:16.289521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:16.289523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:16.289740 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:16.290459 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:09:16.290467 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:09:16.291072 (+ 605us) client_negotiation.cc:769] Sending connection context
0926 03:09:16.291376 (+ 304us) client_negotiation.cc:240] Negotiation successful
0926 03:09:16.291824 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":403,"threads_started":1}
I20260926 03:09:16.294943 14017 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38657 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:09:16 dist-test-slave-hmbw krb5kdc[13863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:09:16.312070 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.295796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38657 (local address 127.0.0.1:52824)
0926 03:09:16.296017 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.296037 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:16.296158 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:16.297474 (+ 1316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:16.297479 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:16.297967 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:09:16.298149 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:16.298160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.299873 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.299878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:16.301078 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:16.301093 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.302039 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.302045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:16.302227 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:16.302925 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:09:16.302948 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:09:16.305502 (+ 2554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:09:16.309188 (+ 3686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:16.309195 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:16.309199 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:16.309669 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:16.310105 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:16.310111 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:16.310115 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:16.310265 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:16.310948 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:09:16.310953 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:09:16.311189 (+ 236us) client_negotiation.cc:769] Sending connection context
0926 03:09:16.311312 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 03:09:16.311648 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 03:09:16.312173 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.295998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52824 (local address 127.0.97.61:38657)
0926 03:09:16.296899 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:16.296908 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:16.296942 (+ 34us) server_negotiation.cc:407] Connection header received
0926 03:09:16.297004 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:16.297012 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:16.297272 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:16.297433 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:16.298338 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.299678 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.301248 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.301873 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.301986 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:16.305751 (+ 3765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:16.305769 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:16.305785 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:16.305824 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:16.309020 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:16.309812 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:16.309821 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:16.309825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:16.309993 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:16.310369 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:16.310374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:16.310378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:16.310801 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:16.311021 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:16.311472 (+ 451us) server_negotiation.cc:299] Negotiation successful
0926 03:09:16.311789 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":708,"thread_start_us":524,"threads_started":1}
I20260926 03:09:16.314733 14017 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:09:16.315968 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.315402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38083 (local address 127.0.0.1:43722)
0926 03:09:16.315634 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.315748 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 03:09:16.316212 13969 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38083: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260926 03:09:16.317965 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.317523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38083 (local address 127.0.0.1:43738)
0926 03:09:16.317698 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.317797 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 03:09:16.318284 14017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38083: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38083: connect: Connection refused (error 111)
I20260926 03:09:16.318758 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:38083
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33607,127.0.97.61:38657,127.0.97.60:38083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:09:16.367666 14017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } attempt: 1
I20260926 03:09:16.368894 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.368421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38083 (local address 127.0.0.1:43744)
0926 03:09:16.368637 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.368737 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 03:09:16.369953 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.369605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38083 (local address 127.0.0.1:43746)
0926 03:09:16.369751 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.369838 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 03:09:16.370220 14017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38083: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38083: connect: Connection refused (error 111)
I20260926 03:09:16.438474 14017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } attempt: 2
I20260926 03:09:16.439846 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.439235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38083 (local address 127.0.0.1:43762)
0926 03:09:16.439499 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.439629 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 03:09:16.441090 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.440621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38083 (local address 127.0.0.1:43778)
0926 03:09:16.440810 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.440921 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260926 03:09:16.441373 14017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38083: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38083: connect: Connection refused (error 111)
I20260926 03:09:16.451759 13943 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38657 } attempt: 5
Sep 26 03:09:16 dist-test-slave-hmbw krb5kdc[13863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:09:16.466245 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.452500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38657 (local address 127.0.0.1:52832)
0926 03:09:16.452714 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.452737 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:16.452865 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:16.453227 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:16.453231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:16.453633 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:09:16.453809 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:16.453818 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.454980 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.454986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:16.456363 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:16.456382 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.457292 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.457299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:16.457523 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:16.458175 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:09:16.458197 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:09:16.460679 (+ 2482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:09:16.463755 (+ 3076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:16.463761 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:16.463766 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:16.464146 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:16.464535 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:16.464540 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:16.464544 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:16.464718 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:16.465260 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:09:16.465265 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:09:16.465437 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 03:09:16.465560 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 03:09:16.465877 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 03:09:16.466248 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.452581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52832 (local address 127.0.97.61:38657)
0926 03:09:16.452767 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:16.452773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:16.452793 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:09:16.453016 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:16.453021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:16.453092 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:16.453177 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:16.453963 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.454796 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.456618 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.457098 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.457209 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:16.460897 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:16.460902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:16.460905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:16.460941 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:16.463583 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:16.464279 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:16.464284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:16.464286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:16.464409 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:16.464827 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:16.464832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:16.464836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:16.465130 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:16.465281 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:16.465696 (+ 415us) server_negotiation.cc:299] Negotiation successful
0926 03:09:16.465927 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:09:16.467774 13943 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:09:16.468873 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.468417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38083 (local address 127.0.0.1:43784)
0926 03:09:16.468640 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.468718 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260926 03:09:16.469765 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.469441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38083 (local address 127.0.0.1:43790)
0926 03:09:16.469584 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.469663 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 03:09:16.469952 13943 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38083: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38083: connect: Connection refused (error 111)
I20260926 03:09:16.521210 13943 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } attempt: 1
I20260926 03:09:16.522604 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.522008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38083 (local address 127.0.0.1:43794)
0926 03:09:16.522243 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.522341 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260926 03:09:16.523661 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.523299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38083 (local address 127.0.0.1:43802)
0926 03:09:16.523460 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.523542 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 03:09:16.523886 13943 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38083: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38083: connect: Connection refused (error 111)
I20260926 03:09:16.532649 14017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } attempt: 3
I20260926 03:09:16.534008 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.533447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38083 (local address 127.0.0.1:43806)
0926 03:09:16.533697 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.533803 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 03:09:16.535274 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.534788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38083 (local address 127.0.0.1:43820)
0926 03:09:16.535061 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.535147 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
W20260926 03:09:16.535517 14017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38083: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38083: connect: Connection refused (error 111)
I20260926 03:09:16.592134 13943 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } attempt: 2
I20260926 03:09:16.593453 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.592932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38083 (local address 127.0.0.1:43834)
0926 03:09:16.593174 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.593261 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 03:09:16.594571 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.594155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38083 (local address 127.0.0.1:43844)
0926 03:09:16.594320 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.594394 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 03:09:16.594782 13943 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38083: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38083: connect: Connection refused (error 111)
W20260926 03:09:16.654665 14024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:09:16.655001 14024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:09:16.655148 14024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:09:16.662846 14024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:09:16.663003 14024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:09:16.663084 14024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:09:16.663167 14024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:09:16.663224 14024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:09:16.674021 14024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33607,127.0.97.61:38657,127.0.97.60:38083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:38083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:09:16.675478 14024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:09:16.676870 14024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 03:09:16.678752 14017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } attempt: 4
I20260926 03:09:16.679875 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.679395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38083 (local address 127.0.0.1:43848)
0926 03:09:16.679639 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.679723 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 03:09:16.681006 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.680613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38083 (local address 127.0.0.1:43856)
0926 03:09:16.680784 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.680892 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 03:09:16.681229 14017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38083: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38083: connect: Connection refused (error 111)
W20260926 03:09:16.689436 14029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:16.689327 14030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:16.689436 14032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:16.691107 14024 server_base.cc:1034] running on GCE node
I20260926 03:09:16.692036 14024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:09:16.693701 14024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:09:16.694944 14024 hybrid_clock.cc:648] HybridClock initialized: now 1790392156694885 us; error 50 us; skew 500 ppm
I20260926 03:09:16.695379 14024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:09:16 dist-test-slave-hmbw krb5kdc[13863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:09:16.698855 14024 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260926 03:09:16.700659 14024 webserver.cc:460] Webserver started at http://127.0.97.60:35905/ using document root <none> and password file <none>
I20260926 03:09:16.700981 13943 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } attempt: 3
I20260926 03:09:16.701532 14024 fs_manager.cc:359] Metadata directory not provided
I20260926 03:09:16.701653 14024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:09:16.702023 14024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:09:16.702157 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.701610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38083 (local address 127.0.0.1:43872)
0926 03:09:16.701855 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.701959 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 03:09:16.703440 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.702950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38083 (local address 127.0.0.1:43882)
0926 03:09:16.703190 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.703287 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260926 03:09:16.703704 13943 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38083: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38083: connect: Connection refused (error 111)
I20260926 03:09:16.706197 14024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-2/data/instance:
uuid: "e9764baad9884fd79f4e3e4ba580d0aa"
format_stamp: "Formatted at 2026-09-26 03:09:16 on dist-test-slave-hmbw"
server_key: "9af3b937789cbbacee2416084378fdc0"
server_key_iv: "27330383d42158b155de81fc344cef81"
server_key_version: "encryptionkey@0"
I20260926 03:09:16.707330 14024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-2/wal/instance:
uuid: "e9764baad9884fd79f4e3e4ba580d0aa"
format_stamp: "Formatted at 2026-09-26 03:09:16 on dist-test-slave-hmbw"
server_key: "9af3b937789cbbacee2416084378fdc0"
server_key_iv: "27330383d42158b155de81fc344cef81"
server_key_version: "encryptionkey@0"
I20260926 03:09:16.713501 14024 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20260926 03:09:16.717681 14039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:16.719376 14024 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:09:16.719578 14024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-2/wal
uuid: "e9764baad9884fd79f4e3e4ba580d0aa"
format_stamp: "Formatted at 2026-09-26 03:09:16 on dist-test-slave-hmbw"
server_key: "9af3b937789cbbacee2416084378fdc0"
server_key_iv: "27330383d42158b155de81fc344cef81"
server_key_version: "encryptionkey@0"
I20260926 03:09:16.719808 14024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:09:16.748857 14024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:09:16.754124 14024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:09:16.754441 14024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:09:16.773245 14024 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:38083
I20260926 03:09:16.773267 14090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:38083 every 8 connection(s)
I20260926 03:09:16.774824 14024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-2/data/info.pb
I20260926 03:09:16.778779 14091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:09:16.781443 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 14024
I20260926 03:09:16.781639 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391950058021-388-0/minicluster-data/master-2/wal/instance
I20260926 03:09:16.782213 388 external_mini_cluster.cc:1428] Setting key b0d9931d52b69186c40e3c226952d7ea
I20260926 03:09:16.787050 14091 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33607 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:09:16 dist-test-slave-hmbw krb5kdc[13863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 03:09:16 dist-test-slave-hmbw krb5kdc[13863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:09:16.806174 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.789683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60994 (local address 127.0.97.62:33607)
0926 03:09:16.789902 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:16.789907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:16.790710 (+ 803us) server_negotiation.cc:407] Connection header received
0926 03:09:16.790958 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:16.790962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:16.791033 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:16.791123 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:16.792411 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.793388 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.795272 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.795711 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.795835 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:16.799614 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:16.799620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:16.799624 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:16.799663 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:16.802361 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:16.803302 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:16.803307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:16.803311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:16.803502 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:16.803959 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:16.803963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:16.803966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:16.804331 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:16.804449 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:16.805475 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0926 03:09:16.805745 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:09:16.807331 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.784310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43888 (local address 127.0.97.60:38083)
0926 03:09:16.785221 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:16.785234 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:16.785277 (+ 43us) server_negotiation.cc:407] Connection header received
0926 03:09:16.786989 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:16.787014 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:16.787460 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:16.787878 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:16.789386 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.791313 (+ 1927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.793285 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.794478 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.794716 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:16.799344 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:16.799356 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:16.799373 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:16.799410 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:16.802459 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:16.803568 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:16.803580 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:16.803583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:16.803824 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:16.804267 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:16.804271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:16.804273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:16.805010 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:16.805311 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:16.805793 (+ 482us) server_negotiation.cc:299] Negotiation successful
0926 03:09:16.806016 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":691,"thread_start_us":440,"threads_started":1}
I20260926 03:09:16.807389 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.789689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33607 (local address 127.0.0.1:60994)
0926 03:09:16.790540 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.790600 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:16.790784 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:16.791201 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:16.791208 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:16.791957 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:09:16.792214 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:16.792223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.793565 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.793570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:16.795113 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:16.795123 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.795954 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.795961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:16.796104 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:16.796841 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:09:16.796869 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:09:16.799316 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:09:16.802655 (+ 3339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:16.802668 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:16.802683 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:16.803135 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:16.803630 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:16.803635 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:16.803638 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:16.803851 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:16.804461 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:09:16.804469 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:09:16.805003 (+ 534us) client_negotiation.cc:769] Sending connection context
0926 03:09:16.805300 (+ 297us) client_negotiation.cc:240] Negotiation successful
0926 03:09:16.805779 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":3,"thread_start_us":377,"threads_started":1}
I20260926 03:09:16.809561 14091 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38657 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:09:16 dist-test-slave-hmbw krb5kdc[13863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 26 03:09:16 dist-test-slave-hmbw krb5kdc[13863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:09:16.823084 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.810378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38657 (local address 127.0.0.1:52848)
0926 03:09:16.810655 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.810672 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:16.810797 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:16.811131 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:16.811136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:16.811576 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:09:16.811749 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:16.811759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.812727 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.812732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:16.813616 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:16.813628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.814319 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.814325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:16.814601 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:16.815189 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:09:16.815207 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:09:16.817498 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:09:16.820563 (+ 3065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:16.820570 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:16.820574 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:16.820989 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:16.821392 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:16.821397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:16.821401 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:16.821549 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:16.822069 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:09:16.822074 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:09:16.822228 (+ 154us) client_negotiation.cc:769] Sending connection context
0926 03:09:16.822347 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 03:09:16.822722 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260926 03:09:16.823109 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.810371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52848 (local address 127.0.97.61:38657)
0926 03:09:16.810597 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:16.810603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:16.810751 (+ 148us) server_negotiation.cc:407] Connection header received
0926 03:09:16.810922 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:16.810928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:16.810993 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:16.811076 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:16.811891 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.812573 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.813762 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.814171 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.814283 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:16.817735 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:16.817740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:16.817744 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:16.817781 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:16.820396 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:16.821124 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:16.821129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:16.821132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:16.821270 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:16.821663 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:16.821666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:16.821669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:16.821946 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:16.822080 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:16.822465 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 03:09:16.822722 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 03:09:16.824224 14091 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:09:16.826429 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.811493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32778 (local address 127.0.97.62:33607)
0926 03:09:16.811683 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:16.811688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:16.811708 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:09:16.811812 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:16.811817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:16.811883 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:16.811994 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:16.813205 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.813961 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.815312 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.815749 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.815864 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:16.820013 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:16.820019 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:16.820022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:16.820064 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:16.822640 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:16.823442 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:16.823447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:16.823450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:16.823604 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:16.824055 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:16.824061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:16.824065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:16.824356 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:16.824635 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:16.825838 (+ 1203us) server_negotiation.cc:299] Negotiation successful
0926 03:09:16.826096 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 03:09:16 dist-test-slave-hmbw krb5kdc[13863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 03:09:16 dist-test-slave-hmbw krb5kdc[13863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:09:16.838745 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.824890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38083 (local address 127.0.0.1:43896)
0926 03:09:16.825052 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.825070 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:16.825154 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:16.825625 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:16.825629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:16.826091 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:09:16.826275 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:16.826287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.827627 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.827633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:16.828775 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:16.828792 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.829644 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.829651 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:16.829851 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:16.830624 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:09:16.830644 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:09:16.832932 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:09:16.836052 (+ 3120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:16.836057 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:16.836062 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:16.836456 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:16.836810 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:16.836814 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:16.836816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:16.836951 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:16.837601 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:09:16.837605 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:09:16.837749 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 03:09:16.837911 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 03:09:16.838253 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 03:09:16.839155 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.824943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43896 (local address 127.0.97.60:38083)
0926 03:09:16.825191 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:16.825197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:16.825212 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:09:16.825284 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:16.825288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:16.825372 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:16.825465 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:16.826570 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.827445 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.828923 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.829475 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.829578 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:16.833138 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:16.833143 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:16.833146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:16.833179 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:16.835898 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:16.836572 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:16.836577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:16.836580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:16.836700 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:16.837063 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:16.837068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:16.837071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:16.837457 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:16.837608 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:16.838027 (+ 419us) server_negotiation.cc:299] Negotiation successful
0926 03:09:16.838599 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":181,"server-negotiator.queue_time_us":143}
I20260926 03:09:16.843772 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.828581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52864 (local address 127.0.97.61:38657)
0926 03:09:16.828748 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:16.828753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:16.828770 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:09:16.828856 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:16.828861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:16.828944 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:16.829049 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:16.830531 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.831306 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.832970 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.833401 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.833524 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:16.837860 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:16.837867 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:16.837872 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:16.837916 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:16.841074 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:16.841807 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:16.841813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:16.841817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:16.841962 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:16.842322 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:16.842328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:16.842331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:16.842656 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:16.842921 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:16.843215 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 03:09:16.843443 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 03:09:16.843840 14091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9764baad9884fd79f4e3e4ba580d0aa: Bootstrap starting.
I20260926 03:09:16.847776 14091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9764baad9884fd79f4e3e4ba580d0aa: Neither blocks nor log segments found. Creating new log.
I20260926 03:09:16.849072 14091 log.cc:824] T 00000000000000000000000000000000 P e9764baad9884fd79f4e3e4ba580d0aa: Log is configured to *not* fsync() on all Append() calls
I20260926 03:09:16.849315 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:09:16.852206 14091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9764baad9884fd79f4e3e4ba580d0aa: No bootstrap required, opened a new log
I20260926 03:09:16.862449 14091 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9764baad9884fd79f4e3e4ba580d0aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33607 } } peers { permanent_uuid: "f017d75d274140819f9092d957859922" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38657 } } peers { permanent_uuid: "e9764baad9884fd79f4e3e4ba580d0aa" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } }
I20260926 03:09:16.863027 14091 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9764baad9884fd79f4e3e4ba580d0aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:09:16.863142 14091 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9764baad9884fd79f4e3e4ba580d0aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9764baad9884fd79f4e3e4ba580d0aa, State: Initialized, Role: FOLLOWER
I20260926 03:09:16.864101 14091 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9764baad9884fd79f4e3e4ba580d0aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33607 } } peers { permanent_uuid: "f017d75d274140819f9092d957859922" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38657 } } peers { permanent_uuid: "e9764baad9884fd79f4e3e4ba580d0aa" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } }
I20260926 03:09:16.867961 14104 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9764baad9884fd79f4e3e4ba580d0aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33607 } } peers { permanent_uuid: "f017d75d274140819f9092d957859922" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38657 } } peers { permanent_uuid: "e9764baad9884fd79f4e3e4ba580d0aa" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } } }
I20260926 03:09:16.868381 14104 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9764baad9884fd79f4e3e4ba580d0aa [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:09:16.869800 14091 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9764baad9884fd79f4e3e4ba580d0aa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:09:16.878067 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.856912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52870 (local address 127.0.97.61:38657)
0926 03:09:16.857122 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:16.857130 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:16.858122 (+ 992us) server_negotiation.cc:407] Connection header received
0926 03:09:16.858678 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:16.858686 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:16.858794 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:16.858922 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:16.861763 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.863048 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.865103 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.865704 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.865829 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:16.870843 (+ 5014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:16.870851 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:16.870856 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:16.870907 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:16.874407 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:16.875479 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:16.875486 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:16.875490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:16.875671 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:16.876194 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:16.876199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:16.876202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:16.876612 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:16.876851 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:16.877430 (+ 579us) server_negotiation.cc:299] Negotiation successful
0926 03:09:16.877690 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:09:16.880939 13943 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } attempt: 4
I20260926 03:09:16.884305 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.856960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32780 (local address 127.0.97.62:33607)
0926 03:09:16.857164 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:16.857171 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:16.864539 (+ 7368us) server_negotiation.cc:407] Connection header received
0926 03:09:16.865115 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:16.865122 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:16.865233 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:16.865310 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:16.867908 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.869211 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.871381 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.872364 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.872490 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:16.876285 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:16.876293 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:16.876298 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:16.876363 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:16.880462 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:16.881337 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:16.881343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:16.881347 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:16.881658 (+ 311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:16.882282 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:16.882287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:16.882291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:16.882758 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:16.883006 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:16.883662 (+ 656us) server_negotiation.cc:299] Negotiation successful
0926 03:09:16.883933 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:09:16.885525 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.858217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43908 (local address 127.0.97.60:38083)
0926 03:09:16.858422 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:16.858427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:16.865272 (+ 6845us) server_negotiation.cc:407] Connection header received
0926 03:09:16.865398 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:16.865404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:16.865489 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:16.865586 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:16.868919 (+ 3333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.870031 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.872336 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.873107 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.873210 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:16.877757 (+ 4547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:16.877765 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:16.877771 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:16.877821 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:16.881902 (+ 4081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:16.882806 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:16.882813 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:16.882816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:16.883087 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:16.883572 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:16.883577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:16.883580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:16.884010 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:16.884251 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:16.884904 (+ 653us) server_negotiation.cc:299] Negotiation successful
0926 03:09:16.885158 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 03:09:16 dist-test-slave-hmbw krb5kdc[13863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260926 03:09:16.898729 14116 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e9764baad9884fd79f4e3e4ba580d0aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:09:16.898890 14116 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:09:16.902293 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.881620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38083 (local address 127.0.0.1:43916)
0926 03:09:16.881841 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.881861 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:16.881974 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:16.886161 (+ 4187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:16.886167 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:16.886688 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:09:16.886904 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:16.886917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.888454 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.888460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:16.889385 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:16.889398 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.890399 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.890405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:16.890659 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:16.891319 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:09:16.891342 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:09:16.893812 (+ 2470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:09:16.899067 (+ 5255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:16.899074 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:16.899078 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:16.899673 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:16.900211 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:16.900216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:16.900219 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:16.900400 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:16.901144 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:09:16.901150 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:09:16.901391 (+ 241us) client_negotiation.cc:769] Sending connection context
0926 03:09:16.901529 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 03:09:16.901862 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 03:09:16.904255 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.882274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43916 (local address 127.0.97.60:38083)
0926 03:09:16.885784 (+ 3510us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:16.885789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:16.885806 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:09:16.885854 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:16.885858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:16.885991 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:16.887045 (+ 1054us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:16.887174 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.888277 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.889552 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.890230 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.890988 (+ 758us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:16.894021 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:16.894026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:16.894029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:16.894060 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:16.898669 (+ 4609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:16.899867 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:16.899872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:16.899875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:16.900052 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:16.900571 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:16.900575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:16.900578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:16.900998 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:16.903365 (+ 2367us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:16.903526 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 03:09:16.903814 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3396,"thread_start_us":5520,"threads_started":1}
I20260926 03:09:16.893855 14024 master_runner.cc:186] Error getting master registration for 127.0.97.62:33607: OK, instance_id { permanent_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" instance_seqno: 1790392155740640 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:09:16.905205 14024 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 03:09:16.909351 13943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1: Bootstrap starting.
I20260926 03:09:16.914220 13943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1: Neither blocks nor log segments found. Creating new log.
I20260926 03:09:16.915412 13943 log.cc:824] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1: Log is configured to *not* fsync() on all Append() calls
I20260926 03:09:16.918390 13943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1: No bootstrap required, opened a new log
I20260926 03:09:16.929997 13943 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33607 } } peers { permanent_uuid: "f017d75d274140819f9092d957859922" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38657 } } peers { permanent_uuid: "e9764baad9884fd79f4e3e4ba580d0aa" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } }
I20260926 03:09:16.930577 13943 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:09:16.930732 13943 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a758ee4c4834b63be4a6ff739daa3b1, State: Initialized, Role: FOLLOWER
I20260926 03:09:16.931491 13943 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33607 } } peers { permanent_uuid: "f017d75d274140819f9092d957859922" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38657 } } peers { permanent_uuid: "e9764baad9884fd79f4e3e4ba580d0aa" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } }
I20260926 03:09:16.932991 14119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33607 } } peers { permanent_uuid: "f017d75d274140819f9092d957859922" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38657 } } peers { permanent_uuid: "e9764baad9884fd79f4e3e4ba580d0aa" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } } }
I20260926 03:09:16.933362 14119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:09:16.934154 13943 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 03:09:16.947496 14130 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:09:16.947665 14130 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:09:16.944376 13876 master_runner.cc:186] Error getting master registration for 127.0.97.61:38657: OK, instance_id { permanent_uuid: "f017d75d274140819f9092d957859922" instance_seqno: 1790392156248502 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:09:16.957679 13876 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 03:09:16.980479 14017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } attempt: 5
W20260926 03:09:16.980787 13897 tablet.cc:2369] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 03:09:16 dist-test-slave-hmbw krb5kdc[13863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 03:09:17.001230 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.981194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38083 (local address 127.0.0.1:43924)
0926 03:09:16.981660 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:16.981682 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:16.981817 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:16.982435 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:16.982442 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:16.983038 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:09:16.983272 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:16.983286 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.985140 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.985149 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:16.986908 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:16.986932 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.988292 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.988299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:16.988574 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:16.989303 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:09:16.989339 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:09:16.992709 (+ 3370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:09:16.997327 (+ 4618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:16.997335 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:16.997339 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:16.997999 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:16.998972 (+ 973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:16.998979 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:16.998982 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:16.999205 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:17.000063 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:09:17.000069 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:09:17.000267 (+ 198us) client_negotiation.cc:769] Sending connection context
0926 03:09:17.000478 (+ 211us) client_negotiation.cc:240] Negotiation successful
0926 03:09:17.000837 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322}
I20260926 03:09:17.002369 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:16.981834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43924 (local address 127.0.97.60:38083)
0926 03:09:16.982020 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:16.982027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:16.982050 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:09:16.982127 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:16.982132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:16.982218 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:16.982347 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:16.983608 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.984807 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.987138 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:16.987852 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:16.988055 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:16.993031 (+ 4976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:16.993039 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:16.993045 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:16.993093 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:16.997063 (+ 3970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:16.998202 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:16.998210 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:16.998215 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:16.998751 (+ 536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:16.999449 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:16.999456 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:16.999459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:16.999902 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:17.000081 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:17.001657 (+ 1576us) server_negotiation.cc:299] Negotiation successful
0926 03:09:17.001955 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 03:09:17.007244 14045 tablet.cc:2369] T 00000000000000000000000000000000 P e9764baad9884fd79f4e3e4ba580d0aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:09:17.009176 14017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f017d75d274140819f9092d957859922: Bootstrap starting.
I20260926 03:09:17.014415 14017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f017d75d274140819f9092d957859922: Neither blocks nor log segments found. Creating new log.
I20260926 03:09:17.016222 14017 log.cc:824] T 00000000000000000000000000000000 P f017d75d274140819f9092d957859922: Log is configured to *not* fsync() on all Append() calls
I20260926 03:09:17.020112 14017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f017d75d274140819f9092d957859922: No bootstrap required, opened a new log
I20260926 03:09:17.027122 14119 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:09:17.027401 14119 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33607 } } peers { permanent_uuid: "f017d75d274140819f9092d957859922" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38657 } } peers { permanent_uuid: "e9764baad9884fd79f4e3e4ba580d0aa" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } }
I20260926 03:09:17.029172 14119 leader_election.cc:288] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f017d75d274140819f9092d957859922 (127.0.97.61:38657), e9764baad9884fd79f4e3e4ba580d0aa (127.0.97.60:38083)
I20260926 03:09:17.030467 14066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9764baad9884fd79f4e3e4ba580d0aa" is_pre_election: true
I20260926 03:09:17.031256 14066 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e9764baad9884fd79f4e3e4ba580d0aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a758ee4c4834b63be4a6ff739daa3b1 in term 0.
I20260926 03:09:17.033084 13895 leader_election.cc:302] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a758ee4c4834b63be4a6ff739daa3b1, e9764baad9884fd79f4e3e4ba580d0aa; no voters:
I20260926 03:09:17.033182 14017 raft_consensus.cc:348] T 00000000000000000000000000000000 P f017d75d274140819f9092d957859922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33607 } } peers { permanent_uuid: "f017d75d274140819f9092d957859922" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38657 } } peers { permanent_uuid: "e9764baad9884fd79f4e3e4ba580d0aa" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } }
I20260926 03:09:17.033665 14017 raft_consensus.cc:374] T 00000000000000000000000000000000 P f017d75d274140819f9092d957859922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:09:17.033667 14119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:09:17.033795 14017 raft_consensus.cc:729] T 00000000000000000000000000000000 P f017d75d274140819f9092d957859922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f017d75d274140819f9092d957859922, State: Initialized, Role: FOLLOWER
I20260926 03:09:17.033844 14119 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:09:17.033959 14119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:09:17.034641 14017 consensus_queue.cc:260] T 00000000000000000000000000000000 P f017d75d274140819f9092d957859922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33607 } } peers { permanent_uuid: "f017d75d274140819f9092d957859922" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38657 } } peers { permanent_uuid: "e9764baad9884fd79f4e3e4ba580d0aa" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } }
I20260926 03:09:17.039739 14133 sys_catalog.cc:455] T 00000000000000000000000000000000 P f017d75d274140819f9092d957859922 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33607 } } peers { permanent_uuid: "f017d75d274140819f9092d957859922" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38657 } } peers { permanent_uuid: "e9764baad9884fd79f4e3e4ba580d0aa" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } } }
I20260926 03:09:17.040208 14133 sys_catalog.cc:458] T 00000000000000000000000000000000 P f017d75d274140819f9092d957859922 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:09:17.041301 14017 sys_catalog.cc:564] T 00000000000000000000000000000000 P f017d75d274140819f9092d957859922 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:09:17.041355 13992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f017d75d274140819f9092d957859922" is_pre_election: true
I20260926 03:09:17.042033 13992 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f017d75d274140819f9092d957859922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a758ee4c4834b63be4a6ff739daa3b1 in term 0.
I20260926 03:09:17.042644 14119 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33607 } } peers { permanent_uuid: "f017d75d274140819f9092d957859922" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38657 } } peers { permanent_uuid: "e9764baad9884fd79f4e3e4ba580d0aa" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } }
I20260926 03:09:17.046139 13992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f017d75d274140819f9092d957859922"
I20260926 03:09:17.046417 13992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f017d75d274140819f9092d957859922 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:09:17.050275 13992 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f017d75d274140819f9092d957859922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a758ee4c4834b63be4a6ff739daa3b1 in term 1.
I20260926 03:09:17.051069 13893 leader_election.cc:302] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a758ee4c4834b63be4a6ff739daa3b1, f017d75d274140819f9092d957859922; no voters:
I20260926 03:09:17.052316 14136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:09:17.052667 14066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9764baad9884fd79f4e3e4ba580d0aa"
I20260926 03:09:17.052910 14066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9764baad9884fd79f4e3e4ba580d0aa [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:09:17.056505 14066 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e9764baad9884fd79f4e3e4ba580d0aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a758ee4c4834b63be4a6ff739daa3b1 in term 1.
I20260926 03:09:17.057102 14136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1 [term 1 LEADER]: Becoming Leader. State: Replica: 5a758ee4c4834b63be4a6ff739daa3b1, State: Running, Role: LEADER
I20260926 03:09:17.057454 14119 leader_election.cc:288] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1 [CANDIDATE]: Term 1 election: Requested vote from peers f017d75d274140819f9092d957859922 (127.0.97.61:38657), e9764baad9884fd79f4e3e4ba580d0aa (127.0.97.60:38083)
I20260926 03:09:17.057860 14136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33607 } } peers { permanent_uuid: "f017d75d274140819f9092d957859922" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38657 } } peers { permanent_uuid: "e9764baad9884fd79f4e3e4ba580d0aa" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } }
I20260926 03:09:17.067876 14137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a758ee4c4834b63be4a6ff739daa3b1. Latest consensus state: current_term: 1 leader_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33607 } } peers { permanent_uuid: "f017d75d274140819f9092d957859922" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38657 } } peers { permanent_uuid: "e9764baad9884fd79f4e3e4ba580d0aa" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } } }
I20260926 03:09:17.068383 14137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1 [sys.catalog]: This master's current role is: LEADER
I20260926 03:09:17.078641 14141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:09:17.081140 14141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:09:17.100019 14066 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e9764baad9884fd79f4e3e4ba580d0aa [term 1 FOLLOWER]: Refusing update from remote peer 5a758ee4c4834b63be4a6ff739daa3b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:09:17.101323 14137 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9764baad9884fd79f4e3e4ba580d0aa" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:09:17.105870 13992 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f017d75d274140819f9092d957859922 [term 1 FOLLOWER]: Refusing update from remote peer 5a758ee4c4834b63be4a6ff739daa3b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:09:17.107110 14137 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f017d75d274140819f9092d957859922" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:09:17.129326 14104 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9764baad9884fd79f4e3e4ba580d0aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a758ee4c4834b63be4a6ff739daa3b1. Latest consensus state: current_term: 1 leader_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33607 } } peers { permanent_uuid: "f017d75d274140819f9092d957859922" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38657 } } peers { permanent_uuid: "e9764baad9884fd79f4e3e4ba580d0aa" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } } }
I20260926 03:09:17.129679 14104 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9764baad9884fd79f4e3e4ba580d0aa [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:09:17.130925 14133 sys_catalog.cc:455] T 00000000000000000000000000000000 P f017d75d274140819f9092d957859922 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a758ee4c4834b63be4a6ff739daa3b1. Latest consensus state: current_term: 1 leader_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33607 } } peers { permanent_uuid: "f017d75d274140819f9092d957859922" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38657 } } peers { permanent_uuid: "e9764baad9884fd79f4e3e4ba580d0aa" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } } }
I20260926 03:09:17.131256 14133 sys_catalog.cc:458] T 00000000000000000000000000000000 P f017d75d274140819f9092d957859922 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:09:17.132584 14142 mvcc.cc:205] Tried to move back new op lower bound from 7333446275468046336 to 7333446275316350976. Current Snapshot: MvccSnapshot[applied={T|T < 7333446275468046336}]
I20260926 03:09:17.137369 14137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33607 } } peers { permanent_uuid: "f017d75d274140819f9092d957859922" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38657 } } peers { permanent_uuid: "e9764baad9884fd79f4e3e4ba580d0aa" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } } }
I20260926 03:09:17.137703 14137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1 [sys.catalog]: This master's current role is: LEADER
I20260926 03:09:17.137969 14136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33607 } } peers { permanent_uuid: "f017d75d274140819f9092d957859922" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38657 } } peers { permanent_uuid: "e9764baad9884fd79f4e3e4ba580d0aa" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } } }
I20260926 03:09:17.138233 14136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1 [sys.catalog]: This master's current role is: LEADER
I20260926 03:09:17.138636 14145 mvcc.cc:205] Tried to move back new op lower bound from 7333446275468046336 to 7333446275316350976. Current Snapshot: MvccSnapshot[applied={T|T < 7333446275468046336}]
I20260926 03:09:17.142637 14148 mvcc.cc:205] Tried to move back new op lower bound from 7333446275468046336 to 7333446275316350976. Current Snapshot: MvccSnapshot[applied={T|T < 7333446275468046336}]
I20260926 03:09:17.143383 14141 catalog_manager.cc:1347] Generated new cluster ID: 6c06c17b6aa64e209d9ef8888fcffa3c
I20260926 03:09:17.143502 14141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:09:17.148909 14104 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9764baad9884fd79f4e3e4ba580d0aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33607 } } peers { permanent_uuid: "f017d75d274140819f9092d957859922" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38657 } } peers { permanent_uuid: "e9764baad9884fd79f4e3e4ba580d0aa" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } } }
I20260926 03:09:17.149230 14104 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9764baad9884fd79f4e3e4ba580d0aa [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:09:17.155537 14133 sys_catalog.cc:455] T 00000000000000000000000000000000 P f017d75d274140819f9092d957859922 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a758ee4c4834b63be4a6ff739daa3b1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33607 } } peers { permanent_uuid: "f017d75d274140819f9092d957859922" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38657 } } peers { permanent_uuid: "e9764baad9884fd79f4e3e4ba580d0aa" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38083 } } }
I20260926 03:09:17.155876 14133 sys_catalog.cc:458] T 00000000000000000000000000000000 P f017d75d274140819f9092d957859922 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:09:17.159461 14153 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 03:09:17.189527 14141 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:09:17.191165 14141 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:09:17.211061 14141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a758ee4c4834b63be4a6ff739daa3b1: Generated new TSK 0
I20260926 03:09:17.212348 14141 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:09:17.900938 14116 catalog_manager.cc:1259] Loaded cluster ID: 6c06c17b6aa64e209d9ef8888fcffa3c
I20260926 03:09:17.901048 14116 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e9764baad9884fd79f4e3e4ba580d0aa: loading cluster ID for follower catalog manager: success
I20260926 03:09:17.903563 14116 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e9764baad9884fd79f4e3e4ba580d0aa: acquiring CA information for follower catalog manager: success
I20260926 03:09:17.905498 14116 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e9764baad9884fd79f4e3e4ba580d0aa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:09:18.163961 14153 catalog_manager.cc:1259] Loaded cluster ID: 6c06c17b6aa64e209d9ef8888fcffa3c
I20260926 03:09:18.164073 14153 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f017d75d274140819f9092d957859922: loading cluster ID for follower catalog manager: success
I20260926 03:09:18.167155 14153 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f017d75d274140819f9092d957859922: acquiring CA information for follower catalog manager: success
I20260926 03:09:18.169399 14153 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f017d75d274140819f9092d957859922: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:09:18.315433 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 13876
I20260926 03:09:18.316614 13938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:09:18.432550 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 13876
I20260926 03:09:18.451433 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 13950
W20260926 03:09:18.451578 13968 connection.cc:535] server connection from 127.0.0.1:52832 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 03:09:18.452522 14012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:09:18.566409 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 13950
I20260926 03:09:18.582612 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 14024
I20260926 03:09:18.583760 14086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:09:18.692343 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 14024
2026-09-26T03:09:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4655 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:09:18 dist-test-slave-hmbw krb5kdc[14169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:09:18 dist-test-slave-hmbw krb5kdc[14169](info): set up 2 sockets
Sep 26 03:09:18 dist-test-slave-hmbw krb5kdc[14169](info): commencing operation
krb5kdc: starting...
W20260926 03:09:19.718796 388 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:09:19 dist-test-slave-hmbw krb5kdc[14169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:09:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:09:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:09:19.950840 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44041
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44041
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:09:20.311177 14182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:09:20.311509 14182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:09:20.311632 14182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:09:20.319195 14182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:09:20.319322 14182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:09:20.319375 14182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:09:20.319464 14182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:09:20.319537 14182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:09:20.329840 14182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:09:20.331211 14182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:09:20.332557 14182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:09:20.344611 14187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:20.344708 14188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:20.344730 14190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:20.345949 14182 server_base.cc:1034] running on GCE node
I20260926 03:09:20.346820 14182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:09:20.348389 14182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:09:20.349668 14182 hybrid_clock.cc:648] HybridClock initialized: now 1790392160349606 us; error 44 us; skew 500 ppm
I20260926 03:09:20.350108 14182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:09:20 dist-test-slave-hmbw krb5kdc[14169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:09:20.353539 14182 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:09:20.355322 14182 webserver.cc:460] Webserver started at http://127.0.97.62:35249/ using document root <none> and password file <none>
I20260926 03:09:20.356055 14182 fs_manager.cc:359] Metadata directory not provided
I20260926 03:09:20.356181 14182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:09:20.356496 14182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:09:20.359781 14182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "c95fdbcb2d8f4f2b95ae9529b714a5d6"
format_stamp: "Formatted at 2026-09-26 03:09:20 on dist-test-slave-hmbw"
server_key: "fdf7627f55f3e138520589a5fa6292c3"
server_key_iv: "064fb6864401b83ed6ff3785a684366e"
server_key_version: "encryptionkey@0"
I20260926 03:09:20.360625 14182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "c95fdbcb2d8f4f2b95ae9529b714a5d6"
format_stamp: "Formatted at 2026-09-26 03:09:20 on dist-test-slave-hmbw"
server_key: "fdf7627f55f3e138520589a5fa6292c3"
server_key_iv: "064fb6864401b83ed6ff3785a684366e"
server_key_version: "encryptionkey@0"
I20260926 03:09:20.366576 14182 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 03:09:20.370842 14197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:20.372706 14182 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260926 03:09:20.372915 14182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "c95fdbcb2d8f4f2b95ae9529b714a5d6"
format_stamp: "Formatted at 2026-09-26 03:09:20 on dist-test-slave-hmbw"
server_key: "fdf7627f55f3e138520589a5fa6292c3"
server_key_iv: "064fb6864401b83ed6ff3785a684366e"
server_key_version: "encryptionkey@0"
I20260926 03:09:20.373131 14182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:09:20.396385 14182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:09:20.400918 14182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:09:20.401199 14182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:09:20.420044 14182 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44041
I20260926 03:09:20.420075 14248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44041 every 8 connection(s)
I20260926 03:09:20.421532 14182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:09:20.424099 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 14182
I20260926 03:09:20.424301 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:09:20.424940 388 external_mini_cluster.cc:1428] Setting key d7dd48557fd9cb12782fa38fd048b8e9
I20260926 03:09:20.425535 14249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:09:20.438935 14249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c95fdbcb2d8f4f2b95ae9529b714a5d6: Bootstrap starting.
Sep 26 03:09:20 dist-test-slave-hmbw krb5kdc[14169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:09:20.442564 14249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c95fdbcb2d8f4f2b95ae9529b714a5d6: Neither blocks nor log segments found. Creating new log.
I20260926 03:09:20.443909 14249 log.cc:824] T 00000000000000000000000000000000 P c95fdbcb2d8f4f2b95ae9529b714a5d6: Log is configured to *not* fsync() on all Append() calls
I20260926 03:09:20.446877 14249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c95fdbcb2d8f4f2b95ae9529b714a5d6: No bootstrap required, opened a new log
I20260926 03:09:20.449611 14252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:20.426866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45810 (local address 127.0.97.62:44041)
0926 03:09:20.427657 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:20.427670 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:20.427714 (+ 44us) server_negotiation.cc:407] Connection header received
0926 03:09:20.430344 (+ 2630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:20.430372 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:20.430917 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:20.431353 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:20.432941 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:20.434312 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:20.435886 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:20.436752 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:20.436965 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:20.441772 (+ 4807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:20.441782 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:20.441799 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:20.441854 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:20.445130 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:20.445959 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:20.445967 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:20.445974 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:20.446143 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:20.446540 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:20.446546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:20.446549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:20.447099 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:20.447399 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:20.447842 (+ 443us) server_negotiation.cc:299] Negotiation successful
0926 03:09:20.448285 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":329,"threads_started":1}
I20260926 03:09:20.453830 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:09:20.455154 14249 raft_consensus.cc:348] T 00000000000000000000000000000000 P c95fdbcb2d8f4f2b95ae9529b714a5d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c95fdbcb2d8f4f2b95ae9529b714a5d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44041 } }
I20260926 03:09:20.455576 14249 raft_consensus.cc:374] T 00000000000000000000000000000000 P c95fdbcb2d8f4f2b95ae9529b714a5d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:09:20.455677 14249 raft_consensus.cc:729] T 00000000000000000000000000000000 P c95fdbcb2d8f4f2b95ae9529b714a5d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c95fdbcb2d8f4f2b95ae9529b714a5d6, State: Initialized, Role: FOLLOWER
I20260926 03:09:20.456331 14249 consensus_queue.cc:260] T 00000000000000000000000000000000 P c95fdbcb2d8f4f2b95ae9529b714a5d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c95fdbcb2d8f4f2b95ae9529b714a5d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44041 } }
I20260926 03:09:20.456561 14249 raft_consensus.cc:388] T 00000000000000000000000000000000 P c95fdbcb2d8f4f2b95ae9529b714a5d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:09:20.456694 14249 raft_consensus.cc:482] T 00000000000000000000000000000000 P c95fdbcb2d8f4f2b95ae9529b714a5d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:09:20.456843 14249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c95fdbcb2d8f4f2b95ae9529b714a5d6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:09:20.459143 14249 raft_consensus.cc:504] T 00000000000000000000000000000000 P c95fdbcb2d8f4f2b95ae9529b714a5d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c95fdbcb2d8f4f2b95ae9529b714a5d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44041 } }
I20260926 03:09:20.459661 14249 leader_election.cc:302] T 00000000000000000000000000000000 P c95fdbcb2d8f4f2b95ae9529b714a5d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c95fdbcb2d8f4f2b95ae9529b714a5d6; no voters:
I20260926 03:09:20.460230 14249 leader_election.cc:288] T 00000000000000000000000000000000 P c95fdbcb2d8f4f2b95ae9529b714a5d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:09:20.460434 14254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c95fdbcb2d8f4f2b95ae9529b714a5d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:09:20.461088 14254 raft_consensus.cc:686] T 00000000000000000000000000000000 P c95fdbcb2d8f4f2b95ae9529b714a5d6 [term 1 LEADER]: Becoming Leader. State: Replica: c95fdbcb2d8f4f2b95ae9529b714a5d6, State: Running, Role: LEADER
I20260926 03:09:20.461611 14254 consensus_queue.cc:237] T 00000000000000000000000000000000 P c95fdbcb2d8f4f2b95ae9529b714a5d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c95fdbcb2d8f4f2b95ae9529b714a5d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44041 } }
I20260926 03:09:20.462363 14249 sys_catalog.cc:564] T 00000000000000000000000000000000 P c95fdbcb2d8f4f2b95ae9529b714a5d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:09:20.464411 14256 sys_catalog.cc:455] T 00000000000000000000000000000000 P c95fdbcb2d8f4f2b95ae9529b714a5d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c95fdbcb2d8f4f2b95ae9529b714a5d6. Latest consensus state: current_term: 1 leader_uuid: "c95fdbcb2d8f4f2b95ae9529b714a5d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c95fdbcb2d8f4f2b95ae9529b714a5d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44041 } } }
I20260926 03:09:20.464394 14255 sys_catalog.cc:455] T 00000000000000000000000000000000 P c95fdbcb2d8f4f2b95ae9529b714a5d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c95fdbcb2d8f4f2b95ae9529b714a5d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c95fdbcb2d8f4f2b95ae9529b714a5d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44041 } } }
I20260926 03:09:20.464831 14256 sys_catalog.cc:458] T 00000000000000000000000000000000 P c95fdbcb2d8f4f2b95ae9529b714a5d6 [sys.catalog]: This master's current role is: LEADER
I20260926 03:09:20.464843 14255 sys_catalog.cc:458] T 00000000000000000000000000000000 P c95fdbcb2d8f4f2b95ae9529b714a5d6 [sys.catalog]: This master's current role is: LEADER
I20260926 03:09:20.465624 14260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:09:20.471683 14260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:09:20.482378 14260 catalog_manager.cc:1347] Generated new cluster ID: 51ddb18eaa4b499284c255ac9d2c5919
I20260926 03:09:20.482537 14260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:09:20.512605 14260 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:09:20.514701 14260 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:09:20.533387 14260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c95fdbcb2d8f4f2b95ae9529b714a5d6: Generated new TSK 0
I20260926 03:09:20.534315 14260 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:09:20.565738 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 14182
I20260926 03:09:20.567011 14244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:09:20.664676 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 14182
2026-09-26T03:09:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1969 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:09:20 dist-test-slave-hmbw krb5kdc[14277](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:09:20 dist-test-slave-hmbw krb5kdc[14277](info): set up 2 sockets
Sep 26 03:09:20 dist-test-slave-hmbw krb5kdc[14277](info): commencing operation
krb5kdc: starting...
W20260926 03:09:21.690851 388 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:09:21 dist-test-slave-hmbw krb5kdc[14277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:09:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:09:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:09:21.926924 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40383
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:09:22.281327 14290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:09:22.281661 14290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:09:22.281778 14290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:09:22.289310 14290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:09:22.289438 14290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:09:22.289490 14290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:09:22.289569 14290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:09:22.289631 14290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:09:22.300086 14290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:09:22.301503 14290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:09:22.302982 14290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:09:22.316133 14296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:22.316136 14295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:22.316673 14298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:22.321549 14290 server_base.cc:1034] running on GCE node
I20260926 03:09:22.322662 14290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:09:22.324620 14290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:09:22.325932 14290 hybrid_clock.cc:648] HybridClock initialized: now 1790392162325822 us; error 63 us; skew 500 ppm
I20260926 03:09:22.326443 14290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:09:22 dist-test-slave-hmbw krb5kdc[14277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:09:22.331682 14290 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:09:22.333650 14290 webserver.cc:460] Webserver started at http://127.0.97.62:42561/ using document root <none> and password file <none>
I20260926 03:09:22.334441 14290 fs_manager.cc:359] Metadata directory not provided
I20260926 03:09:22.334607 14290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:09:22.334928 14290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:09:22.338171 14290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "95a5c9e578354ddbabdef1290ed59933"
format_stamp: "Formatted at 2026-09-26 03:09:22 on dist-test-slave-hmbw"
server_key: "3dc1381332c74c6378afc8bd10529a58"
server_key_iv: "59ff032508fc965e3ef7d7740ddf0528"
server_key_version: "encryptionkey@0"
I20260926 03:09:22.339088 14290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "95a5c9e578354ddbabdef1290ed59933"
format_stamp: "Formatted at 2026-09-26 03:09:22 on dist-test-slave-hmbw"
server_key: "3dc1381332c74c6378afc8bd10529a58"
server_key_iv: "59ff032508fc965e3ef7d7740ddf0528"
server_key_version: "encryptionkey@0"
I20260926 03:09:22.344780 14290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 03:09:22.348670 14305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:22.350236 14290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:09:22.350481 14290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "95a5c9e578354ddbabdef1290ed59933"
format_stamp: "Formatted at 2026-09-26 03:09:22 on dist-test-slave-hmbw"
server_key: "3dc1381332c74c6378afc8bd10529a58"
server_key_iv: "59ff032508fc965e3ef7d7740ddf0528"
server_key_version: "encryptionkey@0"
I20260926 03:09:22.350746 14290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:09:22.365198 14290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:09:22.370487 14290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:09:22.370821 14290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:09:22.389356 14290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40383
I20260926 03:09:22.389389 14356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40383 every 8 connection(s)
I20260926 03:09:22.391002 14290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:09:22.395545 14357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:09:22.401248 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 14290
I20260926 03:09:22.401489 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:09:22.402311 388 external_mini_cluster.cc:1428] Setting key 17eb123918ed66495285e2973a78b072
I20260926 03:09:22.410374 14357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933: Bootstrap starting.
I20260926 03:09:22.413712 14357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933: Neither blocks nor log segments found. Creating new log.
I20260926 03:09:22.414949 14357 log.cc:824] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933: Log is configured to *not* fsync() on all Append() calls
I20260926 03:09:22.418285 14357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933: No bootstrap required, opened a new log
Sep 26 03:09:22 dist-test-slave-hmbw krb5kdc[14277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:09:22.427564 14357 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a5c9e578354ddbabdef1290ed59933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } }
I20260926 03:09:22.428025 14357 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:09:22.428141 14357 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a5c9e578354ddbabdef1290ed59933, State: Initialized, Role: FOLLOWER
I20260926 03:09:22.428820 14357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a5c9e578354ddbabdef1290ed59933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } }
I20260926 03:09:22.429109 14357 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:09:22.429265 14357 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:09:22.429456 14357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:09:22.431496 14360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:22.404675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48554 (local address 127.0.97.62:40383)
0926 03:09:22.408731 (+ 4056us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:22.408750 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:22.409419 (+ 669us) server_negotiation.cc:407] Connection header received
0926 03:09:22.411422 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:22.411448 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:22.411878 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:22.412285 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:22.413806 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:22.415311 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:22.417030 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:22.417834 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:22.417994 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:22.423035 (+ 5041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:22.423044 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:22.423062 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:22.423117 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:22.426569 (+ 3452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:22.427606 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:22.427614 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:22.427622 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:22.427820 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:22.428265 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:22.428271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:22.428274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:22.428887 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:22.429176 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:22.429743 (+ 567us) server_negotiation.cc:299] Negotiation successful
0926 03:09:22.430195 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3765,"thread_start_us":3496,"threads_started":1}
I20260926 03:09:22.432250 14357 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a5c9e578354ddbabdef1290ed59933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } }
I20260926 03:09:22.432830 14357 leader_election.cc:302] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95a5c9e578354ddbabdef1290ed59933; no voters:
I20260926 03:09:22.433624 14357 leader_election.cc:288] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:09:22.433830 14362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:09:22.434824 14362 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [term 1 LEADER]: Becoming Leader. State: Replica: 95a5c9e578354ddbabdef1290ed59933, State: Running, Role: LEADER
I20260926 03:09:22.435523 14362 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a5c9e578354ddbabdef1290ed59933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } }
I20260926 03:09:22.437490 14357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:09:22.438906 14363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95a5c9e578354ddbabdef1290ed59933. Latest consensus state: current_term: 1 leader_uuid: "95a5c9e578354ddbabdef1290ed59933" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a5c9e578354ddbabdef1290ed59933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } } }
I20260926 03:09:22.438954 14364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95a5c9e578354ddbabdef1290ed59933" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a5c9e578354ddbabdef1290ed59933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } } }
I20260926 03:09:22.439407 14364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [sys.catalog]: This master's current role is: LEADER
I20260926 03:09:22.439460 14363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [sys.catalog]: This master's current role is: LEADER
W20260926 03:09:22.449601 14378 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:09:22.449755 14378 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:09:22.449921 14367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:09:22.452122 14367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:09:22.461586 14367 catalog_manager.cc:1347] Generated new cluster ID: ad5e00c37c6d46b0abcd168ed408069d
I20260926 03:09:22.461733 14367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:09:22.486073 14367 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:09:22.487510 14367 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:09:22.505151 14367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933: Generated new TSK 0
I20260926 03:09:22.506304 14367 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:09:22.571422 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40383
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:09:22.902014 14383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:09:22.902304 14383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:09:22.902444 14383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:09:22.909833 14383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:09:22.909962 14383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:09:22.910105 14383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:09:22.920508 14383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:09:22.921926 14383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:09:22.923291 14383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:09:22.937475 14388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:22.937655 14391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:22.937497 14389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:22.938262 14383 server_base.cc:1034] running on GCE node
I20260926 03:09:22.938977 14383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:09:22.940064 14383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:09:22.941301 14383 hybrid_clock.cc:648] HybridClock initialized: now 1790392162941243 us; error 46 us; skew 500 ppm
I20260926 03:09:22.941689 14383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:09:22 dist-test-slave-hmbw krb5kdc[14277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:09:22.945098 14383 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:09:22.946884 14383 webserver.cc:460] Webserver started at http://127.0.97.1:37257/ using document root <none> and password file <none>
I20260926 03:09:22.947615 14383 fs_manager.cc:359] Metadata directory not provided
I20260926 03:09:22.947729 14383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:09:22.948027 14383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:09:22.951138 14383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/data/instance:
uuid: "5d61fa4921724c1abb3d38b989a0aafc"
format_stamp: "Formatted at 2026-09-26 03:09:22 on dist-test-slave-hmbw"
server_key: "b6daab19fae6d01d209882cd78ae2bf4"
server_key_iv: "d008da5ed934ac711aa4ffc6b15d38f9"
server_key_version: "encryptionkey@0"
I20260926 03:09:22.951984 14383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5d61fa4921724c1abb3d38b989a0aafc"
format_stamp: "Formatted at 2026-09-26 03:09:22 on dist-test-slave-hmbw"
server_key: "b6daab19fae6d01d209882cd78ae2bf4"
server_key_iv: "d008da5ed934ac711aa4ffc6b15d38f9"
server_key_version: "encryptionkey@0"
I20260926 03:09:22.957626 14383 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:09:22.961443 14398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:22.963033 14383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 03:09:22.963205 14383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "5d61fa4921724c1abb3d38b989a0aafc"
format_stamp: "Formatted at 2026-09-26 03:09:22 on dist-test-slave-hmbw"
server_key: "b6daab19fae6d01d209882cd78ae2bf4"
server_key_iv: "d008da5ed934ac711aa4ffc6b15d38f9"
server_key_version: "encryptionkey@0"
I20260926 03:09:22.963435 14383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:09:22.987413 14383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:09:22.992540 14383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:09:22.992847 14383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:09:22.993917 14383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:09:22.995424 14383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:09:22.995509 14383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:22.995628 14383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:09:22.995679 14383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:23.028353 14383 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45459
I20260926 03:09:23.028388 14510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45459 every 8 connection(s)
I20260926 03:09:23.029603 14383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:09:23.033774 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 14383
I20260926 03:09:23.034003 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:09:23.034837 388 external_mini_cluster.cc:1428] Setting key 9cf08133d0ccfa370ab2a8e7528401de
Sep 26 03:09:23 dist-test-slave-hmbw krb5kdc[14277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:09:23.053535 14513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:23.032235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58819 (local address 127.0.97.62:40383)
0926 03:09:23.032724 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:23.032729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:23.033626 (+ 897us) server_negotiation.cc:407] Connection header received
0926 03:09:23.036103 (+ 2477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:23.036109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:23.036198 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:23.036299 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:23.038448 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:23.039410 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:23.041729 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:23.042211 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:23.042326 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:23.046928 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:23.046936 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:23.046941 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:23.046987 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:23.050087 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:23.051089 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:23.051095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:23.051099 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:23.051257 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:23.051696 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:23.051700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:23.051703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:23.052040 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:23.052179 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:23.052922 (+ 743us) server_negotiation.cc:299] Negotiation successful
0926 03:09:23.053170 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":291,"threads_started":1}
I20260926 03:09:23.054764 14514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:23.032631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.1:58819)
0926 03:09:23.033413 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:23.033466 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:23.035796 (+ 2330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:23.036590 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:23.036600 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:23.037289 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:09:23.038232 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:23.038261 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:23.039617 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:23.039623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:23.041537 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:23.041557 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:23.042478 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:23.042485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:23.042818 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:23.043648 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:09:23.043680 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:09:23.046639 (+ 2959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:09:23.050324 (+ 3685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:23.050332 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:23.050349 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:23.050956 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:23.051379 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:23.051382 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:23.051384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:23.051582 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:23.052163 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:09:23.052169 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:09:23.052543 (+ 374us) client_negotiation.cc:769] Sending connection context
0926 03:09:23.052780 (+ 237us) client_negotiation.cc:240] Negotiation successful
0926 03:09:23.053220 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":283,"threads_started":1}
I20260926 03:09:23.056757 14511 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40383
I20260926 03:09:23.057116 14511 heartbeater.cc:464] Registering TS with master...
I20260926 03:09:23.057976 14511 heartbeater.cc:511] Master 127.0.97.62:40383 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:09:23.060441 14322 ts_manager.cc:194] Registered new tserver with Master: 5d61fa4921724c1abb3d38b989a0aafc (127.0.97.1:45459)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 03:09:23.062000 14322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:09:23.090528 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40383
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:09:23.422416 14517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:09:23.422744 14517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:09:23.422848 14517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:09:23.430248 14517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:09:23.430387 14517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:09:23.430526 14517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:09:23.440698 14517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:09:23.441987 14517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:09:23.443411 14517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:09:23.456594 14522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:23.456854 14525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:23.457331 14523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:23.460031 14517 server_base.cc:1034] running on GCE node
I20260926 03:09:23.460752 14517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:09:23.461778 14517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:09:23.462988 14517 hybrid_clock.cc:648] HybridClock initialized: now 1790392163462923 us; error 59 us; skew 500 ppm
I20260926 03:09:23.463315 14517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:09:23 dist-test-slave-hmbw krb5kdc[14277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:09:23.466651 14517 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:09:23.468327 14517 webserver.cc:460] Webserver started at http://127.0.97.2:36537/ using document root <none> and password file <none>
I20260926 03:09:23.469045 14517 fs_manager.cc:359] Metadata directory not provided
I20260926 03:09:23.469158 14517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:09:23.469452 14517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:09:23.472469 14517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/data/instance:
uuid: "8ef908b570fb41e78d622d1b09d75a63"
format_stamp: "Formatted at 2026-09-26 03:09:23 on dist-test-slave-hmbw"
server_key: "8785fbe15b78d428044c1f521acdf7ee"
server_key_iv: "c6f284986fb496dddf33ef438350177e"
server_key_version: "encryptionkey@0"
I20260926 03:09:23.473253 14517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8ef908b570fb41e78d622d1b09d75a63"
format_stamp: "Formatted at 2026-09-26 03:09:23 on dist-test-slave-hmbw"
server_key: "8785fbe15b78d428044c1f521acdf7ee"
server_key_iv: "c6f284986fb496dddf33ef438350177e"
server_key_version: "encryptionkey@0"
I20260926 03:09:23.478634 14517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 03:09:23.482239 14532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:23.483613 14517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:09:23.483759 14517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "8ef908b570fb41e78d622d1b09d75a63"
format_stamp: "Formatted at 2026-09-26 03:09:23 on dist-test-slave-hmbw"
server_key: "8785fbe15b78d428044c1f521acdf7ee"
server_key_iv: "c6f284986fb496dddf33ef438350177e"
server_key_version: "encryptionkey@0"
I20260926 03:09:23.483969 14517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:09:23.499037 14517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:09:23.503242 14517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:09:23.503520 14517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:09:23.504498 14517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:09:23.506019 14517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:09:23.506096 14517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:23.506225 14517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:09:23.506278 14517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:23.536890 14517 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36847
I20260926 03:09:23.536945 14644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36847 every 8 connection(s)
I20260926 03:09:23.538300 14517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:09:23.543355 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 14517
I20260926 03:09:23.543514 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:09:23.543946 388 external_mini_cluster.cc:1428] Setting key adafd1cb7152fe022e66357830e7ddc4
Sep 26 03:09:23 dist-test-slave-hmbw krb5kdc[14277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:09:23.559554 14513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:23.540948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57727 (local address 127.0.97.62:40383)
0926 03:09:23.541120 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:23.541124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:23.542328 (+ 1204us) server_negotiation.cc:407] Connection header received
0926 03:09:23.544400 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:23.544406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:23.544495 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:23.544583 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:23.546126 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:23.546911 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:23.548546 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:23.548991 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:23.549104 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:23.553137 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:23.553143 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:23.553146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:23.553183 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:23.556071 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:23.557102 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:23.557106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:23.557109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:23.557245 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:23.557753 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:23.557757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:23.557760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:23.558083 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:23.558210 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:23.559113 (+ 903us) server_negotiation.cc:299] Negotiation successful
0926 03:09:23.559317 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 03:09:23.561002 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:23.541330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.2:57727)
0926 03:09:23.542097 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:23.542141 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:23.544121 (+ 1980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:23.544831 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:23.544841 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:23.545375 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:09:23.545946 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:23.545964 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:23.547076 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:23.547080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:23.548392 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:23.548405 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:23.549393 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:23.549404 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:23.549635 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:23.550400 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:09:23.550426 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:09:23.552822 (+ 2396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:09:23.556276 (+ 3454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:23.556287 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:23.556309 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:23.556960 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:23.557351 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:23.557354 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:23.557357 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:23.557605 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:23.558219 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:09:23.558226 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:09:23.558720 (+ 494us) client_negotiation.cc:769] Sending connection context
0926 03:09:23.559016 (+ 296us) client_negotiation.cc:240] Negotiation successful
0926 03:09:23.559584 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":285,"threads_started":1}
I20260926 03:09:23.562573 14645 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40383
I20260926 03:09:23.562939 14645 heartbeater.cc:464] Registering TS with master...
I20260926 03:09:23.563930 14645 heartbeater.cc:511] Master 127.0.97.62:40383 requested a full tablet report, sending...
I20260926 03:09:23.565673 14322 ts_manager.cc:194] Registered new tserver with Master: 8ef908b570fb41e78d622d1b09d75a63 (127.0.97.2:36847)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 03:09:23.566589 14322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:09:23.594602 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40383
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:09:23.925109 14650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:09:23.925406 14650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:09:23.925500 14650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:09:23.932832 14650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:09:23.932953 14650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:09:23.933076 14650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:09:23.943408 14650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:09:23.944669 14650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:09:23.945942 14650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:09:23.959534 14655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:23.959507 14658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:23.959474 14656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:23.960210 14650 server_base.cc:1034] running on GCE node
I20260926 03:09:23.960827 14650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:09:23.961800 14650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:09:23.963009 14650 hybrid_clock.cc:648] HybridClock initialized: now 1790392163962955 us; error 39 us; skew 500 ppm
I20260926 03:09:23.963326 14650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:09:23 dist-test-slave-hmbw krb5kdc[14277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:09:23.966598 14650 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:09:23.968169 14650 webserver.cc:460] Webserver started at http://127.0.97.3:44513/ using document root <none> and password file <none>
I20260926 03:09:23.968889 14650 fs_manager.cc:359] Metadata directory not provided
I20260926 03:09:23.969007 14650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:09:23.969288 14650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:09:23.972239 14650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/data/instance:
uuid: "c92c5a7d5a7944329f0026645a4253bb"
format_stamp: "Formatted at 2026-09-26 03:09:23 on dist-test-slave-hmbw"
server_key: "23f80321cad9451b9b6eef0cdaa13f5e"
server_key_iv: "bf4fd59836a5ce9055317d5d11bcbb5f"
server_key_version: "encryptionkey@0"
I20260926 03:09:23.972976 14650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c92c5a7d5a7944329f0026645a4253bb"
format_stamp: "Formatted at 2026-09-26 03:09:23 on dist-test-slave-hmbw"
server_key: "23f80321cad9451b9b6eef0cdaa13f5e"
server_key_iv: "bf4fd59836a5ce9055317d5d11bcbb5f"
server_key_version: "encryptionkey@0"
I20260926 03:09:23.978444 14650 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:09:23.982096 14665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:23.983637 14650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:09:23.983793 14650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "c92c5a7d5a7944329f0026645a4253bb"
format_stamp: "Formatted at 2026-09-26 03:09:23 on dist-test-slave-hmbw"
server_key: "23f80321cad9451b9b6eef0cdaa13f5e"
server_key_iv: "bf4fd59836a5ce9055317d5d11bcbb5f"
server_key_version: "encryptionkey@0"
I20260926 03:09:23.983994 14650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:09:24.005371 14650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:09:24.010188 14650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:09:24.010481 14650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:09:24.011561 14650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:09:24.012979 14650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:09:24.013052 14650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:24.013166 14650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:09:24.013216 14650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:24.044202 14650 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43181
I20260926 03:09:24.044257 14777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43181 every 8 connection(s)
I20260926 03:09:24.045526 14650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:09:24.047196 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 14650
I20260926 03:09:24.047371 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:09:24.047787 388 external_mini_cluster.cc:1428] Setting key 09d2290be0f36f31b144c526f08b1574
Sep 26 03:09:24 dist-test-slave-hmbw krb5kdc[14277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:09:24.065482 14511 heartbeater.cc:503] Master 127.0.97.62:40383 was elected leader, sending a full tablet report...
I20260926 03:09:24.067858 14513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:24.048939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42431 (local address 127.0.97.62:40383)
0926 03:09:24.049112 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:24.049119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:24.050775 (+ 1656us) server_negotiation.cc:407] Connection header received
0926 03:09:24.052750 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:24.052754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:24.052831 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:24.052935 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:24.054734 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.055560 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.057357 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.057770 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.057897 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:24.061521 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:24.061528 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:24.061533 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:24.061588 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:24.064420 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:24.065172 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:24.065177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:24.065179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:24.065358 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:24.066012 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:24.066016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:24.066018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:24.066417 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:24.066583 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:24.067390 (+ 807us) server_negotiation.cc:299] Negotiation successful
0926 03:09:24.067569 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:09:24.068847 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:24.049516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.3:42431)
0926 03:09:24.050593 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:24.050655 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:24.052445 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:24.053206 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:24.053215 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:24.053768 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:09:24.054551 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:24.054578 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.055729 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.055733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:24.057127 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:24.057141 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.058001 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.058006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:24.058216 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:24.058916 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:09:24.058944 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:09:24.061246 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:09:24.064565 (+ 3319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:24.064573 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:24.064588 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:24.065036 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:24.065474 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:24.065478 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:24.065481 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:24.065726 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:24.066576 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:09:24.066582 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:09:24.066964 (+ 382us) client_negotiation.cc:769] Sending connection context
0926 03:09:24.067269 (+ 305us) client_negotiation.cc:240] Negotiation successful
0926 03:09:24.067807 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":394,"threads_started":1}
I20260926 03:09:24.070192 14778 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40383
I20260926 03:09:24.070456 14778 heartbeater.cc:464] Registering TS with master...
I20260926 03:09:24.071177 14778 heartbeater.cc:511] Master 127.0.97.62:40383 requested a full tablet report, sending...
I20260926 03:09:24.072570 14322 ts_manager.cc:194] Registered new tserver with Master: c92c5a7d5a7944329f0026645a4253bb (127.0.97.3:43181)
I20260926 03:09:24.073401 14322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42431
I20260926 03:09:24.075975 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:09:24.093741 14513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:24.080444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48558 (local address 127.0.97.62:40383)
0926 03:09:24.080628 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:24.080634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:24.080934 (+ 300us) server_negotiation.cc:407] Connection header received
0926 03:09:24.081100 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:24.081105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:24.081167 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:24.081249 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:24.082617 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.083316 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.084635 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.085072 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.085183 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:24.087729 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:24.087734 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:24.087737 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:24.087783 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:24.090855 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:24.091624 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:24.091629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:24.091631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:24.091790 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:24.092191 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:24.092195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:24.092197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:24.092539 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:24.092807 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:24.093152 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 03:09:24.093416 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:09:24.098676 14322 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:09:24.105582 14322 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:09:24.129664 14790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:24.118162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45459 (local address 127.0.0.1:50620)
0926 03:09:24.119112 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:24.119163 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:24.119352 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:24.120696 (+ 1344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:24.120716 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:24.120752 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:09:24.121157 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:24.121169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.123606 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.123613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:24.126242 (+ 2629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:24.126262 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.128526 (+ 2264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.128534 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:24.128793 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:24.128866 (+ 73us) client_negotiation.cc:769] Sending connection context
0926 03:09:24.128997 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 03:09:24.129221 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":1195,"threads_started":1}
I20260926 03:09:24.131831 14791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:24.118632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50620 (local address 127.0.97.1:45459)
0926 03:09:24.120020 (+ 1388us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:24.120030 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:24.120061 (+ 31us) server_negotiation.cc:407] Connection header received
0926 03:09:24.120169 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:24.120187 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:24.120440 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:24.121388 (+ 948us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:09:24.121639 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.123408 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.126551 (+ 3143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.128255 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.130574 (+ 2319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:24.130854 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:24.130932 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 03:09:24.131098 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1223,"thread_start_us":1082,"threads_started":1}
I20260926 03:09:24.138360 14793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:24.119040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46408 (local address 127.0.97.3:43181)
0926 03:09:24.130688 (+ 11648us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:24.130697 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:24.130735 (+ 38us) server_negotiation.cc:407] Connection header received
0926 03:09:24.130841 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:24.130850 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:24.131134 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:24.131370 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:09:24.132337 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.134280 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.136408 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.137369 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.137492 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:24.137725 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:24.137986 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 03:09:24.138112 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11438,"thread_start_us":12551,"threads_started":1}
I20260926 03:09:24.138432 14790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:24.118791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43181 (local address 127.0.0.1:46408)
0926 03:09:24.129986 (+ 11195us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:24.130008 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:24.130134 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:24.131350 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:24.131355 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:24.131386 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:09:24.131571 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:24.131582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.134428 (+ 2846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.134433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:24.136228 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:24.136242 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.137534 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.137540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:24.137801 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:24.137837 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 03:09:24.137943 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 03:09:24.138086 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11080,"mutex_wait_us":68,"thread_start_us":13083,"threads_started":1}
I20260926 03:09:24.138249 14446 tablet_service.cc:1467] Processing CreateTablet for tablet 3db0e1c7f63447809d8d2986023bd5a5 (DEFAULT_TABLE table=test-table [id=5c60dfeb8ff94a63b60e46824894a917]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:09:24.139983 14446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3db0e1c7f63447809d8d2986023bd5a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:09:24.141935 14713 tablet_service.cc:1467] Processing CreateTablet for tablet 3db0e1c7f63447809d8d2986023bd5a5 (DEFAULT_TABLE table=test-table [id=5c60dfeb8ff94a63b60e46824894a917]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:09:24.143352 14713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3db0e1c7f63447809d8d2986023bd5a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:09:24.144096 14792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:24.120163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36847 (local address 127.0.0.1:49412)
0926 03:09:24.132176 (+ 12013us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:24.132195 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:24.132320 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:24.133467 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:24.133473 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:24.133508 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:09:24.133989 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:24.134003 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.135609 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.135615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:24.138704 (+ 3089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:24.138719 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.143232 (+ 4513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.143239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:24.143412 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:24.143450 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 03:09:24.143543 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 03:09:24.143696 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11891,"mutex_wait_us":1,"thread_start_us":13617,"threads_started":1}
I20260926 03:09:24.145395 14795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:24.120683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49412 (local address 127.0.97.2:36847)
0926 03:09:24.132809 (+ 12126us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:24.132818 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:24.132849 (+ 31us) server_negotiation.cc:407] Connection header received
0926 03:09:24.132945 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:24.132954 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:24.133239 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:24.133445 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:09:24.134190 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.135452 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.138851 (+ 3399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.142956 (+ 4105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.143104 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:24.144793 (+ 1689us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:24.144869 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 03:09:24.145048 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11957,"thread_start_us":11854,"threads_started":1}
I20260926 03:09:24.149562 14580 tablet_service.cc:1467] Processing CreateTablet for tablet 3db0e1c7f63447809d8d2986023bd5a5 (DEFAULT_TABLE table=test-table [id=5c60dfeb8ff94a63b60e46824894a917]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:09:24.151394 14580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3db0e1c7f63447809d8d2986023bd5a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:09:24.154454 14796 tablet_bootstrap.cc:492] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc: Bootstrap starting.
I20260926 03:09:24.157282 14797 tablet_bootstrap.cc:492] T 3db0e1c7f63447809d8d2986023bd5a5 P c92c5a7d5a7944329f0026645a4253bb: Bootstrap starting.
I20260926 03:09:24.159327 14796 tablet_bootstrap.cc:654] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc: Neither blocks nor log segments found. Creating new log.
I20260926 03:09:24.160269 14797 tablet_bootstrap.cc:654] T 3db0e1c7f63447809d8d2986023bd5a5 P c92c5a7d5a7944329f0026645a4253bb: Neither blocks nor log segments found. Creating new log.
I20260926 03:09:24.160638 14796 log.cc:824] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc: Log is configured to *not* fsync() on all Append() calls
I20260926 03:09:24.161180 14797 log.cc:824] T 3db0e1c7f63447809d8d2986023bd5a5 P c92c5a7d5a7944329f0026645a4253bb: Log is configured to *not* fsync() on all Append() calls
I20260926 03:09:24.163969 14796 tablet_bootstrap.cc:492] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc: No bootstrap required, opened a new log
I20260926 03:09:24.164212 14797 tablet_bootstrap.cc:492] T 3db0e1c7f63447809d8d2986023bd5a5 P c92c5a7d5a7944329f0026645a4253bb: No bootstrap required, opened a new log
I20260926 03:09:24.164279 14796 ts_tablet_manager.cc:1397] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 03:09:24.164462 14797 ts_tablet_manager.cc:1397] T 3db0e1c7f63447809d8d2986023bd5a5 P c92c5a7d5a7944329f0026645a4253bb: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 03:09:24.166152 14800 tablet_bootstrap.cc:492] T 3db0e1c7f63447809d8d2986023bd5a5 P 8ef908b570fb41e78d622d1b09d75a63: Bootstrap starting.
I20260926 03:09:24.169421 14800 tablet_bootstrap.cc:654] T 3db0e1c7f63447809d8d2986023bd5a5 P 8ef908b570fb41e78d622d1b09d75a63: Neither blocks nor log segments found. Creating new log.
I20260926 03:09:24.170472 14800 log.cc:824] T 3db0e1c7f63447809d8d2986023bd5a5 P 8ef908b570fb41e78d622d1b09d75a63: Log is configured to *not* fsync() on all Append() calls
I20260926 03:09:24.174074 14800 tablet_bootstrap.cc:492] T 3db0e1c7f63447809d8d2986023bd5a5 P 8ef908b570fb41e78d622d1b09d75a63: No bootstrap required, opened a new log
I20260926 03:09:24.174346 14800 ts_tablet_manager.cc:1397] T 3db0e1c7f63447809d8d2986023bd5a5 P 8ef908b570fb41e78d622d1b09d75a63: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 03:09:24.174875 14797 raft_consensus.cc:348] T 3db0e1c7f63447809d8d2986023bd5a5 P c92c5a7d5a7944329f0026645a4253bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d61fa4921724c1abb3d38b989a0aafc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45459 } } peers { permanent_uuid: "8ef908b570fb41e78d622d1b09d75a63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36847 } } peers { permanent_uuid: "c92c5a7d5a7944329f0026645a4253bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43181 } }
I20260926 03:09:24.174875 14796 raft_consensus.cc:348] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d61fa4921724c1abb3d38b989a0aafc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45459 } } peers { permanent_uuid: "8ef908b570fb41e78d622d1b09d75a63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36847 } } peers { permanent_uuid: "c92c5a7d5a7944329f0026645a4253bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43181 } }
I20260926 03:09:24.175370 14797 raft_consensus.cc:374] T 3db0e1c7f63447809d8d2986023bd5a5 P c92c5a7d5a7944329f0026645a4253bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:09:24.175468 14797 raft_consensus.cc:729] T 3db0e1c7f63447809d8d2986023bd5a5 P c92c5a7d5a7944329f0026645a4253bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92c5a7d5a7944329f0026645a4253bb, State: Initialized, Role: FOLLOWER
I20260926 03:09:24.175477 14796 raft_consensus.cc:374] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:09:24.175570 14796 raft_consensus.cc:729] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d61fa4921724c1abb3d38b989a0aafc, State: Initialized, Role: FOLLOWER
I20260926 03:09:24.176147 14797 consensus_queue.cc:260] T 3db0e1c7f63447809d8d2986023bd5a5 P c92c5a7d5a7944329f0026645a4253bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d61fa4921724c1abb3d38b989a0aafc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45459 } } peers { permanent_uuid: "8ef908b570fb41e78d622d1b09d75a63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36847 } } peers { permanent_uuid: "c92c5a7d5a7944329f0026645a4253bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43181 } }
I20260926 03:09:24.176146 14796 consensus_queue.cc:260] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d61fa4921724c1abb3d38b989a0aafc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45459 } } peers { permanent_uuid: "8ef908b570fb41e78d622d1b09d75a63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36847 } } peers { permanent_uuid: "c92c5a7d5a7944329f0026645a4253bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43181 } }
I20260926 03:09:24.177377 14778 heartbeater.cc:503] Master 127.0.97.62:40383 was elected leader, sending a full tablet report...
I20260926 03:09:24.179631 14797 ts_tablet_manager.cc:1428] T 3db0e1c7f63447809d8d2986023bd5a5 P c92c5a7d5a7944329f0026645a4253bb: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260926 03:09:24.185002 14796 ts_tablet_manager.cc:1428] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc: Time spent starting tablet: real 0.021s user 0.010s sys 0.010s
I20260926 03:09:24.186324 14800 raft_consensus.cc:348] T 3db0e1c7f63447809d8d2986023bd5a5 P 8ef908b570fb41e78d622d1b09d75a63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d61fa4921724c1abb3d38b989a0aafc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45459 } } peers { permanent_uuid: "8ef908b570fb41e78d622d1b09d75a63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36847 } } peers { permanent_uuid: "c92c5a7d5a7944329f0026645a4253bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43181 } }
I20260926 03:09:24.186787 14800 raft_consensus.cc:374] T 3db0e1c7f63447809d8d2986023bd5a5 P 8ef908b570fb41e78d622d1b09d75a63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:09:24.186900 14800 raft_consensus.cc:729] T 3db0e1c7f63447809d8d2986023bd5a5 P 8ef908b570fb41e78d622d1b09d75a63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ef908b570fb41e78d622d1b09d75a63, State: Initialized, Role: FOLLOWER
I20260926 03:09:24.187542 14800 consensus_queue.cc:260] T 3db0e1c7f63447809d8d2986023bd5a5 P 8ef908b570fb41e78d622d1b09d75a63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d61fa4921724c1abb3d38b989a0aafc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45459 } } peers { permanent_uuid: "8ef908b570fb41e78d622d1b09d75a63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36847 } } peers { permanent_uuid: "c92c5a7d5a7944329f0026645a4253bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43181 } }
I20260926 03:09:24.188721 14645 heartbeater.cc:503] Master 127.0.97.62:40383 was elected leader, sending a full tablet report...
I20260926 03:09:24.189252 14800 ts_tablet_manager.cc:1428] T 3db0e1c7f63447809d8d2986023bd5a5 P 8ef908b570fb41e78d622d1b09d75a63: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260926 03:09:24.211556 14803 raft_consensus.cc:482] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:09:24.211892 14803 raft_consensus.cc:504] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d61fa4921724c1abb3d38b989a0aafc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45459 } } peers { permanent_uuid: "8ef908b570fb41e78d622d1b09d75a63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36847 } } peers { permanent_uuid: "c92c5a7d5a7944329f0026645a4253bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43181 } }
I20260926 03:09:24.213779 14803 leader_election.cc:288] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ef908b570fb41e78d622d1b09d75a63 (127.0.97.2:36847), c92c5a7d5a7944329f0026645a4253bb (127.0.97.3:43181)
I20260926 03:09:24.220706 14795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:24.214305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44761 (local address 127.0.97.2:36847)
0926 03:09:24.214536 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:24.214542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:24.215327 (+ 785us) server_negotiation.cc:407] Connection header received
0926 03:09:24.215497 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:24.215503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:24.215580 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:24.215656 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:09:24.216287 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.217067 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.218949 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.219756 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.219884 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:24.219984 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:24.220292 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 03:09:24.220435 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:09:24.220737 14793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:24.214315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42101 (local address 127.0.97.3:43181)
0926 03:09:24.214539 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:24.214545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:24.215469 (+ 924us) server_negotiation.cc:407] Connection header received
0926 03:09:24.215638 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:24.215643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:24.215720 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:24.215810 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:09:24.216404 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.217135 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.218983 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.219753 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.219864 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:24.219976 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:24.220359 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 03:09:24.220489 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:09:24.220839 14805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:24.214236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36847 (local address 127.0.97.1:44761)
0926 03:09:24.215173 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:24.215206 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:24.215364 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:24.215726 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:24.215730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:24.215755 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:09:24.216134 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:24.216144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.217198 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.217203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:24.218798 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:24.218811 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.219998 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.220004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:24.220140 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:24.220164 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 03:09:24.220242 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 03:09:24.220368 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":2,"thread_start_us":552,"threads_started":1}
I20260926 03:09:24.221046 14806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:24.214618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43181 (local address 127.0.97.1:42101)
0926 03:09:24.215356 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:24.215377 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:24.215502 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:24.215845 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:24.215849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:24.215882 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:09:24.216229 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:24.216239 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.217284 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.217287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:24.218825 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:24.218838 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.220074 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.220080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:24.220196 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:24.220219 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 03:09:24.220311 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 03:09:24.220434 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":24,"thread_start_us":420,"threads_started":1}
I20260926 03:09:24.221871 14600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db0e1c7f63447809d8d2986023bd5a5" candidate_uuid: "5d61fa4921724c1abb3d38b989a0aafc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ef908b570fb41e78d622d1b09d75a63" is_pre_election: true
I20260926 03:09:24.221959 14733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db0e1c7f63447809d8d2986023bd5a5" candidate_uuid: "5d61fa4921724c1abb3d38b989a0aafc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92c5a7d5a7944329f0026645a4253bb" is_pre_election: true
I20260926 03:09:24.222352 14600 raft_consensus.cc:2454] T 3db0e1c7f63447809d8d2986023bd5a5 P 8ef908b570fb41e78d622d1b09d75a63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d61fa4921724c1abb3d38b989a0aafc in term 0.
I20260926 03:09:24.222417 14733 raft_consensus.cc:2454] T 3db0e1c7f63447809d8d2986023bd5a5 P c92c5a7d5a7944329f0026645a4253bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d61fa4921724c1abb3d38b989a0aafc in term 0.
I20260926 03:09:24.222998 14400 leader_election.cc:302] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d61fa4921724c1abb3d38b989a0aafc, 8ef908b570fb41e78d622d1b09d75a63; no voters:
I20260926 03:09:24.223380 14803 raft_consensus.cc:2781] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:09:24.223515 14803 raft_consensus.cc:482] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:09:24.223621 14803 raft_consensus.cc:3037] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:09:24.226040 14803 raft_consensus.cc:504] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d61fa4921724c1abb3d38b989a0aafc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45459 } } peers { permanent_uuid: "8ef908b570fb41e78d622d1b09d75a63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36847 } } peers { permanent_uuid: "c92c5a7d5a7944329f0026645a4253bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43181 } }
I20260926 03:09:24.226891 14803 leader_election.cc:288] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc [CANDIDATE]: Term 1 election: Requested vote from peers 8ef908b570fb41e78d622d1b09d75a63 (127.0.97.2:36847), c92c5a7d5a7944329f0026645a4253bb (127.0.97.3:43181)
I20260926 03:09:24.227231 14600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db0e1c7f63447809d8d2986023bd5a5" candidate_uuid: "5d61fa4921724c1abb3d38b989a0aafc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ef908b570fb41e78d622d1b09d75a63"
I20260926 03:09:24.227300 14733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db0e1c7f63447809d8d2986023bd5a5" candidate_uuid: "5d61fa4921724c1abb3d38b989a0aafc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92c5a7d5a7944329f0026645a4253bb"
I20260926 03:09:24.227505 14733 raft_consensus.cc:3037] T 3db0e1c7f63447809d8d2986023bd5a5 P c92c5a7d5a7944329f0026645a4253bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:09:24.227504 14600 raft_consensus.cc:3037] T 3db0e1c7f63447809d8d2986023bd5a5 P 8ef908b570fb41e78d622d1b09d75a63 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:09:24.229810 14600 raft_consensus.cc:2454] T 3db0e1c7f63447809d8d2986023bd5a5 P 8ef908b570fb41e78d622d1b09d75a63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d61fa4921724c1abb3d38b989a0aafc in term 1.
I20260926 03:09:24.229825 14733 raft_consensus.cc:2454] T 3db0e1c7f63447809d8d2986023bd5a5 P c92c5a7d5a7944329f0026645a4253bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d61fa4921724c1abb3d38b989a0aafc in term 1.
I20260926 03:09:24.230366 14400 leader_election.cc:302] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5d61fa4921724c1abb3d38b989a0aafc, 8ef908b570fb41e78d622d1b09d75a63, c92c5a7d5a7944329f0026645a4253bb; no voters:
I20260926 03:09:24.230732 14803 raft_consensus.cc:2781] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:09:24.231191 14803 raft_consensus.cc:686] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc [term 1 LEADER]: Becoming Leader. State: Replica: 5d61fa4921724c1abb3d38b989a0aafc, State: Running, Role: LEADER
I20260926 03:09:24.231742 14803 consensus_queue.cc:237] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d61fa4921724c1abb3d38b989a0aafc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45459 } } peers { permanent_uuid: "8ef908b570fb41e78d622d1b09d75a63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36847 } } peers { permanent_uuid: "c92c5a7d5a7944329f0026645a4253bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43181 } }
I20260926 03:09:24.236509 14321 catalog_manager.cc:5495] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d61fa4921724c1abb3d38b989a0aafc (127.0.97.1). New cstate: current_term: 1 leader_uuid: "5d61fa4921724c1abb3d38b989a0aafc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d61fa4921724c1abb3d38b989a0aafc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ef908b570fb41e78d622d1b09d75a63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92c5a7d5a7944329f0026645a4253bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43181 } health_report { overall_health: UNKNOWN } } }
I20260926 03:09:24.269590 14791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:24.264281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50628 (local address 127.0.97.1:45459)
0926 03:09:24.264524 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:24.264530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:24.264549 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:09:24.264675 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:24.264679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:24.264757 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:24.264899 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:09:24.265263 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.265993 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.267461 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.267920 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.268029 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:24.268524 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:09:24.269013 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:09:24.269145 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:24.269257 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 03:09:24.269351 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260926 03:09:24.278100 14600 raft_consensus.cc:1261] T 3db0e1c7f63447809d8d2986023bd5a5 P 8ef908b570fb41e78d622d1b09d75a63 [term 1 FOLLOWER]: Refusing update from remote peer 5d61fa4921724c1abb3d38b989a0aafc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:09:24.278100 14733 raft_consensus.cc:1261] T 3db0e1c7f63447809d8d2986023bd5a5 P c92c5a7d5a7944329f0026645a4253bb [term 1 FOLLOWER]: Refusing update from remote peer 5d61fa4921724c1abb3d38b989a0aafc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:09:24.279132 14803 consensus_queue.cc:1035] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef908b570fb41e78d622d1b09d75a63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:09:24.279521 14807 consensus_queue.cc:1035] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92c5a7d5a7944329f0026645a4253bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 03:09:24.282009 14512 tablet.cc:2369] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:09:24.290185 14646 tablet.cc:2369] T 3db0e1c7f63447809d8d2986023bd5a5 P 8ef908b570fb41e78d622d1b09d75a63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:09:24.291321 14810 mvcc.cc:205] Tried to move back new op lower bound from 7333446304871878656 to 7333446304697552896. Current Snapshot: MvccSnapshot[applied={T|T < 7333446304871878656}]
I20260926 03:09:24.293567 14811 mvcc.cc:205] Tried to move back new op lower bound from 7333446304871878656 to 7333446304697552896. Current Snapshot: MvccSnapshot[applied={T|T < 7333446304871878656}]
I20260926 03:09:24.304370 14321 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:09:24.304628 14321 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:09:24.308604 14321 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933: Sending DeleteTablet for 3 replicas of tablet 3db0e1c7f63447809d8d2986023bd5a5
I20260926 03:09:24.309963 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 14383
I20260926 03:09:24.309899 14713 tablet_service.cc:1514] Processing DeleteTablet for tablet 3db0e1c7f63447809d8d2986023bd5a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:09:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46408
I20260926 03:09:24.310214 14446 tablet_service.cc:1514] Processing DeleteTablet for tablet 3db0e1c7f63447809d8d2986023bd5a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:09:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50620
I20260926 03:09:24.310704 14580 tablet_service.cc:1514] Processing DeleteTablet for tablet 3db0e1c7f63447809d8d2986023bd5a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:09:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49412
I20260926 03:09:24.322711 14821 tablet_replica.cc:330] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc: stopping tablet replica
I20260926 03:09:24.323499 14821 raft_consensus.cc:2229] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:09:24.324276 14821 raft_consensus.cc:2258] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:09:24.326678 14821 ts_tablet_manager.cc:1905] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:09:24.334538 14821 ts_tablet_manager.cc:1918] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:09:24.334851 14821 log.cc:1196] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/wal/wals/3db0e1c7f63447809d8d2986023bd5a5
I20260926 03:09:24.335566 14821 ts_tablet_manager.cc:1939] T 3db0e1c7f63447809d8d2986023bd5a5 P 5d61fa4921724c1abb3d38b989a0aafc: Deleting consensus metadata
I20260926 03:09:24.337786 14307 catalog_manager.cc:4841] TS 5d61fa4921724c1abb3d38b989a0aafc (127.0.97.1:45459): tablet 3db0e1c7f63447809d8d2986023bd5a5 (table test-table [id=5c60dfeb8ff94a63b60e46824894a917]) successfully deleted
I20260926 03:09:24.339010 14822 tablet_replica.cc:330] T 3db0e1c7f63447809d8d2986023bd5a5 P 8ef908b570fb41e78d622d1b09d75a63: stopping tablet replica
I20260926 03:09:24.339556 14822 raft_consensus.cc:2229] T 3db0e1c7f63447809d8d2986023bd5a5 P 8ef908b570fb41e78d622d1b09d75a63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:09:24.340117 14822 raft_consensus.cc:2258] T 3db0e1c7f63447809d8d2986023bd5a5 P 8ef908b570fb41e78d622d1b09d75a63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:09:24.343561 14820 tablet_replica.cc:330] T 3db0e1c7f63447809d8d2986023bd5a5 P c92c5a7d5a7944329f0026645a4253bb: stopping tablet replica
I20260926 03:09:24.344020 14820 raft_consensus.cc:2229] T 3db0e1c7f63447809d8d2986023bd5a5 P c92c5a7d5a7944329f0026645a4253bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:09:24.344532 14820 raft_consensus.cc:2258] T 3db0e1c7f63447809d8d2986023bd5a5 P c92c5a7d5a7944329f0026645a4253bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:09:24.344977 14822 ts_tablet_manager.cc:1905] T 3db0e1c7f63447809d8d2986023bd5a5 P 8ef908b570fb41e78d622d1b09d75a63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:09:24.346472 14820 ts_tablet_manager.cc:1905] T 3db0e1c7f63447809d8d2986023bd5a5 P c92c5a7d5a7944329f0026645a4253bb: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 03:09:24 dist-test-slave-hmbw krb5kdc[14277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:09:24.351922 14822 ts_tablet_manager.cc:1918] T 3db0e1c7f63447809d8d2986023bd5a5 P 8ef908b570fb41e78d622d1b09d75a63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:09:24.352095 14822 log.cc:1196] T 3db0e1c7f63447809d8d2986023bd5a5 P 8ef908b570fb41e78d622d1b09d75a63: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/wal/wals/3db0e1c7f63447809d8d2986023bd5a5
I20260926 03:09:24.352602 14822 ts_tablet_manager.cc:1939] T 3db0e1c7f63447809d8d2986023bd5a5 P 8ef908b570fb41e78d622d1b09d75a63: Deleting consensus metadata
I20260926 03:09:24.353194 14820 ts_tablet_manager.cc:1918] T 3db0e1c7f63447809d8d2986023bd5a5 P c92c5a7d5a7944329f0026645a4253bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:09:24.353354 14820 log.cc:1196] T 3db0e1c7f63447809d8d2986023bd5a5 P c92c5a7d5a7944329f0026645a4253bb: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/wal/wals/3db0e1c7f63447809d8d2986023bd5a5
I20260926 03:09:24.353845 14820 ts_tablet_manager.cc:1939] T 3db0e1c7f63447809d8d2986023bd5a5 P c92c5a7d5a7944329f0026645a4253bb: Deleting consensus metadata
I20260926 03:09:24.354570 14307 catalog_manager.cc:4841] TS 8ef908b570fb41e78d622d1b09d75a63 (127.0.97.2:36847): tablet 3db0e1c7f63447809d8d2986023bd5a5 (table test-table [id=5c60dfeb8ff94a63b60e46824894a917]) successfully deleted
I20260926 03:09:24.355680 14308 catalog_manager.cc:4841] TS c92c5a7d5a7944329f0026645a4253bb (127.0.97.3:43181): tablet 3db0e1c7f63447809d8d2986023bd5a5 (table test-table [id=5c60dfeb8ff94a63b60e46824894a917]) successfully deleted
I20260926 03:09:24.357889 14791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:24.311989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50630 (local address 127.0.97.1:45459)
0926 03:09:24.318255 (+ 6266us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:24.318263 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:24.335577 (+ 17314us) server_negotiation.cc:407] Connection header received
0926 03:09:24.336110 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:24.336117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:24.336210 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:24.336298 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:24.338440 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.339689 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.342681 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.343405 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.343536 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:24.349391 (+ 5855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:24.349416 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:24.349436 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:24.349489 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:24.354821 (+ 5332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:24.355791 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:24.355800 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:24.355803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:24.355997 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:24.356379 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:24.356382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:24.356385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:24.356739 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:24.357016 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:24.357333 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 03:09:24.357577 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6053}
I20260926 03:09:24.358687 14506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:09:24.459817 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 14383
I20260926 03:09:24.474787 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 14517
Sep 26 03:09:24 dist-test-slave-hmbw krb5kdc[14277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:09:24.489751 14795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:24.475574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53114 (local address 127.0.97.2:36847)
0926 03:09:24.475763 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:24.475768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:24.475788 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:09:24.475905 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:24.475909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:24.475975 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:24.476066 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:24.477301 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.478104 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.479478 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.479895 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.480003 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:24.484179 (+ 4176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:24.484195 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:24.484211 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:24.484249 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:24.487146 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:24.487881 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:24.487889 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:24.487891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:24.488021 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:24.488322 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:24.488324 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:24.488326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:24.488728 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:24.489084 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:24.489297 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 03:09:24.489485 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:09:24.490599 14640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:09:24.579873 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 14517
I20260926 03:09:24.593796 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 14650
Sep 26 03:09:24 dist-test-slave-hmbw krb5kdc[14277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:09:24.610071 14793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:24.594617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55990 (local address 127.0.97.3:43181)
0926 03:09:24.594767 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:24.594771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:24.594789 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:09:24.594994 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:24.594999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:24.595072 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:24.595171 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:24.596523 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.597306 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.598800 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:24.599231 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:24.599344 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:24.603990 (+ 4646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:24.604007 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:24.604024 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:24.604066 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:24.606990 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:24.607850 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:24.607859 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:24.607861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:24.608035 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:24.608478 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:24.608482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:24.608484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:24.608910 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:24.609147 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:24.609606 (+ 459us) server_negotiation.cc:299] Negotiation successful
0926 03:09:24.609811 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 03:09:24.610862 14773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:09:24.704972 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 14650
I20260926 03:09:24.718672 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 14290
I20260926 03:09:24.719864 14352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:09:24.831785 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 14290
I20260926 03:09:24.849752 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40383
--webserver_interface=127.0.97.62
--webserver_port=42561
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:09:25.189996 14828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:09:25.190325 14828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:09:25.190449 14828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:09:25.198089 14828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:09:25.198220 14828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:09:25.198298 14828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:09:25.198378 14828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:09:25.198444 14828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:09:25.209403 14828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=42561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:09:25.210793 14828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:09:25.212196 14828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:09:25.234608 14834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:25.234613 14835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:25.234611 14837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:25.235509 14828 server_base.cc:1034] running on GCE node
I20260926 03:09:25.236380 14828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:09:25.238010 14828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:09:25.239228 14828 hybrid_clock.cc:648] HybridClock initialized: now 1790392165239133 us; error 85 us; skew 500 ppm
I20260926 03:09:25.239699 14828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:09:25 dist-test-slave-hmbw krb5kdc[14277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:09:25.243286 14828 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:09:25.245083 14828 webserver.cc:460] Webserver started at http://127.0.97.62:42561/ using document root <none> and password file <none>
I20260926 03:09:25.245869 14828 fs_manager.cc:359] Metadata directory not provided
I20260926 03:09:25.246011 14828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:09:25.252386 14828 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 03:09:25.255723 14844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:25.257251 14828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:09:25.257434 14828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "95a5c9e578354ddbabdef1290ed59933"
format_stamp: "Formatted at 2026-09-26 03:09:22 on dist-test-slave-hmbw"
server_key: "3dc1381332c74c6378afc8bd10529a58"
server_key_iv: "59ff032508fc965e3ef7d7740ddf0528"
server_key_version: "encryptionkey@0"
I20260926 03:09:25.258594 14828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:09:25.287490 14828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:09:25.291918 14828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:09:25.292191 14828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:09:25.310691 14828 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40383
I20260926 03:09:25.310714 14895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40383 every 8 connection(s)
I20260926 03:09:25.312245 14828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:09:25.313387 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 14828
I20260926 03:09:25.314363 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45459
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:40383
--webserver_port=37257
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:09:25.321795 14896 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:09:25.325388 14896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933: Bootstrap starting.
I20260926 03:09:25.345435 14896 log.cc:824] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933: Log is configured to *not* fsync() on all Append() calls
I20260926 03:09:25.357923 14896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:09:25.358479 14896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933: Bootstrap complete.
I20260926 03:09:25.367916 14896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a5c9e578354ddbabdef1290ed59933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } }
I20260926 03:09:25.368795 14896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a5c9e578354ddbabdef1290ed59933, State: Initialized, Role: FOLLOWER
I20260926 03:09:25.369421 14896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a5c9e578354ddbabdef1290ed59933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } }
I20260926 03:09:25.369652 14896 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:09:25.369763 14896 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:09:25.369925 14896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:09:25.373984 14896 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a5c9e578354ddbabdef1290ed59933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } }
I20260926 03:09:25.374558 14896 leader_election.cc:302] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95a5c9e578354ddbabdef1290ed59933; no voters:
I20260926 03:09:25.375161 14896 leader_election.cc:288] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 03:09:25.375319 14900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 03:09:25.376581 14900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [term 2 LEADER]: Becoming Leader. State: Replica: 95a5c9e578354ddbabdef1290ed59933, State: Running, Role: LEADER
I20260926 03:09:25.377230 14900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a5c9e578354ddbabdef1290ed59933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } }
I20260926 03:09:25.377812 14896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:09:25.379848 14902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95a5c9e578354ddbabdef1290ed59933. Latest consensus state: current_term: 2 leader_uuid: "95a5c9e578354ddbabdef1290ed59933" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a5c9e578354ddbabdef1290ed59933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } } }
I20260926 03:09:25.379953 14901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "95a5c9e578354ddbabdef1290ed59933" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a5c9e578354ddbabdef1290ed59933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } } }
I20260926 03:09:25.380224 14902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [sys.catalog]: This master's current role is: LEADER
I20260926 03:09:25.380306 14901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [sys.catalog]: This master's current role is: LEADER
I20260926 03:09:25.390914 14907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:09:25.399565 14907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:09:25.401227 14907 catalog_manager.cc:1259] Loaded cluster ID: ad5e00c37c6d46b0abcd168ed408069d
I20260926 03:09:25.401340 14907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:09:25.404439 14907 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 03:09:25.421505 14907 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 03:09:25.432112 14907 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:09:25.433952 14907 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933: Loaded TSK: 0
I20260926 03:09:25.435856 14907 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 03:09:25.673355 14898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:09:25.673656 14898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:09:25.673799 14898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:09:25.681872 14898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:09:25.682021 14898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:09:25.682214 14898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:09:25.692684 14898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45459
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=37257
--tserver_master_addrs=127.0.97.62:40383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:09:25.694100 14898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:09:25.695528 14898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:09:25.711135 14923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:25.711161 14924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:25.712608 14926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:25.712993 14898 server_base.cc:1034] running on GCE node
I20260926 03:09:25.713681 14898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:09:25.714869 14898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:09:25.716137 14898 hybrid_clock.cc:648] HybridClock initialized: now 1790392165716061 us; error 69 us; skew 500 ppm
I20260926 03:09:25.716576 14898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:09:25 dist-test-slave-hmbw krb5kdc[14277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:09:25.720276 14898 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:09:25.721922 14898 webserver.cc:460] Webserver started at http://127.0.97.1:37257/ using document root <none> and password file <none>
I20260926 03:09:25.722702 14898 fs_manager.cc:359] Metadata directory not provided
I20260926 03:09:25.722851 14898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:09:25.728930 14898 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 03:09:25.732161 14933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:25.733841 14898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:09:25.734026 14898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "5d61fa4921724c1abb3d38b989a0aafc"
format_stamp: "Formatted at 2026-09-26 03:09:22 on dist-test-slave-hmbw"
server_key: "b6daab19fae6d01d209882cd78ae2bf4"
server_key_iv: "d008da5ed934ac711aa4ffc6b15d38f9"
server_key_version: "encryptionkey@0"
I20260926 03:09:25.735203 14898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:09:25.760660 14898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:09:25.765766 14898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:09:25.766038 14898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:09:25.767151 14898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:09:25.768580 14898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:09:25.768676 14898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:25.768787 14898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:09:25.768854 14898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:25.801491 14898 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45459
I20260926 03:09:25.801539 15045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45459 every 8 connection(s)
I20260926 03:09:25.803004 14898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:09:25.809764 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 14898
I20260926 03:09:25.810488 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:36847
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:40383
--webserver_port=36537
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:09:25 dist-test-slave-hmbw krb5kdc[14277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:09:25.830808 15049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:25.806282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.1:49691)
0926 03:09:25.807163 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:25.807214 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:25.809570 (+ 2356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:25.812295 (+ 2725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:25.812306 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:25.813050 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:09:25.813847 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:25.813873 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:25.816376 (+ 2503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:25.816381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:25.817703 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:25.817721 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:25.819020 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:25.819029 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:25.819296 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:25.819986 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:09:25.820015 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:09:25.822594 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:09:25.826137 (+ 3543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:25.826153 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:25.826166 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:25.826716 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:25.827189 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:25.827192 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:25.827268 (+ 76us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:25.827458 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:25.828234 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:09:25.828242 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:09:25.828558 (+ 316us) client_negotiation.cc:769] Sending connection context
0926 03:09:25.828797 (+ 239us) client_negotiation.cc:240] Negotiation successful
0926 03:09:25.829377 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":378,"threads_started":1}
I20260926 03:09:25.830808 15048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:25.806052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49691 (local address 127.0.97.62:40383)
0926 03:09:25.806662 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:25.806675 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:25.807374 (+ 699us) server_negotiation.cc:407] Connection header received
0926 03:09:25.810536 (+ 3162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:25.810559 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:25.811066 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:25.811520 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:25.814172 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:25.816118 (+ 1946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:25.817952 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:25.818735 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:25.818939 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:25.822923 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:25.822932 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:25.822949 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:25.822994 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:25.825923 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:25.826865 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:25.826873 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:25.826879 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:25.827058 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:25.827579 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:25.827583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:25.827585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:25.828077 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:25.828255 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:25.829045 (+ 790us) server_negotiation.cc:299] Negotiation successful
0926 03:09:25.829442 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":240,"threads_started":1}
I20260926 03:09:25.833722 15046 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40383
I20260926 03:09:25.834008 15046 heartbeater.cc:464] Registering TS with master...
I20260926 03:09:25.834827 15046 heartbeater.cc:511] Master 127.0.97.62:40383 requested a full tablet report, sending...
I20260926 03:09:25.837296 14861 ts_manager.cc:194] Registered new tserver with Master: 5d61fa4921724c1abb3d38b989a0aafc (127.0.97.1:45459)
I20260926 03:09:25.838853 14861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49691
W20260926 03:09:26.146344 15050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:09:26.146706 15050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:09:26.146859 15050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:09:26.154448 15050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:09:26.154621 15050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:09:26.154773 15050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:09:26.165110 15050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:36847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36537
--tserver_master_addrs=127.0.97.62:40383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:09:26.166482 15050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:09:26.167855 15050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:09:26.181425 15055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:26.181458 15056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:26.181496 15058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:26.182353 15050 server_base.cc:1034] running on GCE node
I20260926 03:09:26.183007 15050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:09:26.184010 15050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:09:26.185266 15050 hybrid_clock.cc:648] HybridClock initialized: now 1790392166185218 us; error 44 us; skew 500 ppm
I20260926 03:09:26.185676 15050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:09:26 dist-test-slave-hmbw krb5kdc[14277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:09:26.189170 15050 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:09:26.190822 15050 webserver.cc:460] Webserver started at http://127.0.97.2:36537/ using document root <none> and password file <none>
I20260926 03:09:26.191588 15050 fs_manager.cc:359] Metadata directory not provided
I20260926 03:09:26.191728 15050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:09:26.197613 15050 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 03:09:26.200848 15065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:26.202567 15050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:09:26.202760 15050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "8ef908b570fb41e78d622d1b09d75a63"
format_stamp: "Formatted at 2026-09-26 03:09:23 on dist-test-slave-hmbw"
server_key: "8785fbe15b78d428044c1f521acdf7ee"
server_key_iv: "c6f284986fb496dddf33ef438350177e"
server_key_version: "encryptionkey@0"
I20260926 03:09:26.203819 15050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:09:26.223511 15050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:09:26.228220 15050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:09:26.228505 15050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:09:26.229664 15050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:09:26.231192 15050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:09:26.231329 15050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:26.231436 15050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:09:26.231508 15050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:26.261956 15050 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36847
I20260926 03:09:26.262104 15177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36847 every 8 connection(s)
I20260926 03:09:26.263592 15050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:09:26.265288 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 15050
I20260926 03:09:26.265887 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:43181
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:40383
--webserver_port=44513
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:09:26 dist-test-slave-hmbw krb5kdc[14277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:09:26.284837 15048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:26.266422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38885 (local address 127.0.97.62:40383)
0926 03:09:26.266608 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:26.266613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:26.268071 (+ 1458us) server_negotiation.cc:407] Connection header received
0926 03:09:26.270070 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:26.270077 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:26.270169 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:26.270259 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:26.272211 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.272916 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.274603 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.275036 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.275172 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:26.279149 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:26.279155 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:26.279158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:26.279196 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:26.281681 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:26.282488 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:26.282493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:26.282521 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:26.282654 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:26.283091 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:26.283096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:26.283098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:26.283385 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:26.283528 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:26.284249 (+ 721us) server_negotiation.cc:299] Negotiation successful
0926 03:09:26.284493 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 03:09:26.285836 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:26.267104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.2:38885)
0926 03:09:26.267887 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:26.267933 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:26.269734 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:26.270708 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:26.270718 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:26.271357 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:09:26.271989 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:26.272012 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.273096 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.273101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:26.274412 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:26.274428 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.275295 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.275303 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:26.275554 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:26.276370 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:09:26.276410 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:09:26.278881 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:09:26.281854 (+ 2973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:26.281865 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:26.281886 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:26.282351 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:26.282764 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:26.282770 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:26.282774 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:26.282975 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:26.283510 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:09:26.283518 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:09:26.283877 (+ 359us) client_negotiation.cc:769] Sending connection context
0926 03:09:26.284126 (+ 249us) client_negotiation.cc:240] Negotiation successful
0926 03:09:26.284679 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":311,"threads_started":1}
I20260926 03:09:26.287261 15178 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40383
I20260926 03:09:26.287552 15178 heartbeater.cc:464] Registering TS with master...
I20260926 03:09:26.288260 15178 heartbeater.cc:511] Master 127.0.97.62:40383 requested a full tablet report, sending...
I20260926 03:09:26.289722 14861 ts_manager.cc:194] Registered new tserver with Master: 8ef908b570fb41e78d622d1b09d75a63 (127.0.97.2:36847)
I20260926 03:09:26.290589 14861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38885
W20260926 03:09:26.598681 15181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:09:26.598980 15181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:09:26.599102 15181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:09:26.606616 15181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:09:26.606775 15181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:09:26.606937 15181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:09:26.617233 15181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:43181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=44513
--tserver_master_addrs=127.0.97.62:40383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:09:26.618475 15181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:09:26.619765 15181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:09:26.633538 15186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:26.633551 15187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:26.633486 15189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:26.634531 15181 server_base.cc:1034] running on GCE node
I20260926 03:09:26.635210 15181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:09:26.636277 15181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:09:26.637496 15181 hybrid_clock.cc:648] HybridClock initialized: now 1790392166637446 us; error 46 us; skew 500 ppm
I20260926 03:09:26.637828 15181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:09:26 dist-test-slave-hmbw krb5kdc[14277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:09:26.641304 15181 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:09:26.642935 15181 webserver.cc:460] Webserver started at http://127.0.97.3:44513/ using document root <none> and password file <none>
I20260926 03:09:26.643647 15181 fs_manager.cc:359] Metadata directory not provided
I20260926 03:09:26.643781 15181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:09:26.649367 15181 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 03:09:26.652242 15196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:26.653679 15181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260926 03:09:26.653846 15181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "c92c5a7d5a7944329f0026645a4253bb"
format_stamp: "Formatted at 2026-09-26 03:09:23 on dist-test-slave-hmbw"
server_key: "23f80321cad9451b9b6eef0cdaa13f5e"
server_key_iv: "bf4fd59836a5ce9055317d5d11bcbb5f"
server_key_version: "encryptionkey@0"
I20260926 03:09:26.654883 15181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:09:26.669977 15181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:09:26.675709 15181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:09:26.675985 15181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:09:26.677021 15181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:09:26.678596 15181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:09:26.678706 15181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:26.678815 15181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:09:26.678887 15181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:26.710646 15181 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43181
I20260926 03:09:26.710659 15308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43181 every 8 connection(s)
I20260926 03:09:26.712226 15181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:09:26.719291 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 15181
Sep 26 03:09:26 dist-test-slave-hmbw krb5kdc[14277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:09:26.734866 15048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:26.714908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36785 (local address 127.0.97.62:40383)
0926 03:09:26.715139 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:26.715159 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:26.716354 (+ 1195us) server_negotiation.cc:407] Connection header received
0926 03:09:26.718305 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:26.718310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:26.718397 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:26.718529 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:26.720337 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.721359 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.723087 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.723563 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.723676 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:26.727603 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:26.727608 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:26.727611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:26.727643 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:26.730973 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:26.731818 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:26.731823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:26.731827 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:26.731945 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:26.732476 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:26.732485 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:26.732488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:26.732941 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:26.733053 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:26.734020 (+ 967us) server_negotiation.cc:299] Negotiation successful
0926 03:09:26.734545 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":274,"server-negotiator.queue_time_us":84,"spinlock_wait_cycles":5888}
I20260926 03:09:26.734890 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:26.720358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41046 (local address 127.0.97.62:40383)
0926 03:09:26.721150 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:26.721154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:26.721168 (+ 14us) server_negotiation.cc:407] Connection header received
0926 03:09:26.721211 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:26.721215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:26.721268 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:26.721425 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:26.722900 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.723740 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.725120 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.725708 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.725837 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:26.728682 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:26.728689 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:26.728694 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:26.728761 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:26.731693 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:26.732480 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:26.732485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:26.732488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:26.732687 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:26.733080 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:26.733085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:26.733088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:26.733416 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:26.733603 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:26.734025 (+ 422us) server_negotiation.cc:299] Negotiation successful
0926 03:09:26.734379 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":672,"spinlock_wait_cycles":1408,"thread_start_us":529,"threads_started":1}
I20260926 03:09:26.735967 15311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:26.715449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.3:36785)
0926 03:09:26.716170 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:26.716215 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:26.717881 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:26.718758 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:26.718766 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:26.719369 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:09:26.720111 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:26.720142 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.721541 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.721545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:26.722938 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:26.722952 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.723800 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.723808 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:26.724050 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:26.724720 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:09:26.724743 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:09:26.727372 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:09:26.731139 (+ 3767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:26.731147 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:26.731163 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:26.731679 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:26.732051 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:26.732056 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:26.732059 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:26.732327 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:26.733074 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:09:26.733083 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:09:26.733511 (+ 428us) client_negotiation.cc:769] Sending connection context
0926 03:09:26.733839 (+ 328us) client_negotiation.cc:240] Negotiation successful
0926 03:09:26.734407 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":290,"threads_started":1}
I20260926 03:09:26.737748 15309 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40383
I20260926 03:09:26.738056 15309 heartbeater.cc:464] Registering TS with master...
I20260926 03:09:26.738830 15309 heartbeater.cc:511] Master 127.0.97.62:40383 requested a full tablet report, sending...
I20260926 03:09:26.740440 14861 ts_manager.cc:194] Registered new tserver with Master: c92c5a7d5a7944329f0026645a4253bb (127.0.97.3:43181)
I20260926 03:09:26.741250 14861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36785
I20260926 03:09:26.750385 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:09:26.757388 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:26.751294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41062 (local address 127.0.97.62:40383)
0926 03:09:26.751463 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:26.751467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:26.751883 (+ 416us) server_negotiation.cc:407] Connection header received
0926 03:09:26.752062 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:26.752066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:26.752218 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:26.752408 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:09:26.752755 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.753498 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.754991 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.755531 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.755648 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:26.756111 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:09:26.756729 (+ 618us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:09:26.756849 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:26.757041 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 03:09:26.757120 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 03:09:26.758422 14861 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:09:26.765955 14861 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:09:26.792217 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:26.778314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55994 (local address 127.0.97.3:43181)
0926 03:09:26.778939 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:26.778949 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:26.779799 (+ 850us) server_negotiation.cc:407] Connection header received
0926 03:09:26.780067 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:26.780073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:26.782671 (+ 2598us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:26.783959 (+ 1288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:09:26.784267 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.786367 (+ 2100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.789500 (+ 3133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.790828 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.790962 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:26.791213 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:26.791692 (+ 479us) server_negotiation.cc:299] Negotiation successful
0926 03:09:26.791891 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":382,"threads_started":1}
I20260926 03:09:26.793082 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:26.778249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43181 (local address 127.0.0.1:55994)
0926 03:09:26.779643 (+ 1394us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:26.779682 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:26.779869 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:26.782967 (+ 3098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:26.782977 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:26.783015 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:09:26.783488 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:26.783502 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.786614 (+ 3112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.786623 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:26.789292 (+ 2669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:26.789307 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.791061 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.791071 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:26.791408 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:26.791487 (+ 79us) client_negotiation.cc:769] Sending connection context
0926 03:09:26.791635 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 03:09:26.792616 (+ 981us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1201,"mutex_wait_us":65,"thread_start_us":9166,"threads_started":1}
I20260926 03:09:26.796036 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:26.778647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59342 (local address 127.0.97.1:45459)
0926 03:09:26.786523 (+ 7876us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:26.786533 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:26.786569 (+ 36us) server_negotiation.cc:407] Connection header received
0926 03:09:26.786685 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:26.786693 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:26.787002 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:26.787230 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:09:26.788861 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.790290 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.793265 (+ 2975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.794934 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.795061 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:26.795396 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:26.795564 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 03:09:26.795729 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7703,"thread_start_us":7579,"threads_started":1}
I20260926 03:09:26.796061 15323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:26.778949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45459 (local address 127.0.0.1:59342)
0926 03:09:26.780823 (+ 1874us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:26.780845 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:26.780972 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:26.787763 (+ 6791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:26.787768 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:26.787847 (+ 79us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:09:26.788269 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:26.788281 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.790469 (+ 2188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.790475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:26.793028 (+ 2553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:26.793049 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.795177 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.795184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:26.795367 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:26.795401 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 03:09:26.795502 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 03:09:26.795651 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1710,"mutex_wait_us":155,"thread_start_us":369,"threads_started":1}
I20260926 03:09:26.797515 15319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:26.780214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36847 (local address 127.0.0.1:53128)
0926 03:09:26.787390 (+ 7176us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:26.787416 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:26.787545 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:26.789444 (+ 1899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:26.789448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:26.789477 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:09:26.789831 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:26.789843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.791525 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.791530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:26.794402 (+ 2872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:26.794422 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.796698 (+ 2276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.796704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:26.796916 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:26.796952 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 03:09:26.797044 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 03:09:26.797184 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6750,"thread_start_us":885,"threads_started":1}
I20260926 03:09:26.798246 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:26.780561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53128 (local address 127.0.97.2:36847)
0926 03:09:26.788780 (+ 8219us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:26.788789 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:26.788836 (+ 47us) server_negotiation.cc:407] Connection header received
0926 03:09:26.788942 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:26.788952 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:26.789223 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:26.789449 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:09:26.790022 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.791306 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.794717 (+ 3411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.796205 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.796348 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:26.796535 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:26.797790 (+ 1255us) server_negotiation.cc:299] Negotiation successful
0926 03:09:26.797977 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8061,"thread_start_us":7935,"threads_started":1}
I20260926 03:09:26.797828 15244 tablet_service.cc:1467] Processing CreateTablet for tablet b7f3b609fb8042408ee785d19c6745a9 (DEFAULT_TABLE table=test-table [id=bc8fd25d2b7e43e88d0ed9aba264df6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:09:26.799248 15244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7f3b609fb8042408ee785d19c6745a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:09:26.800113 14981 tablet_service.cc:1467] Processing CreateTablet for tablet b7f3b609fb8042408ee785d19c6745a9 (DEFAULT_TABLE table=test-table [id=bc8fd25d2b7e43e88d0ed9aba264df6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:09:26.801694 14981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7f3b609fb8042408ee785d19c6745a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:09:26.805078 15113 tablet_service.cc:1467] Processing CreateTablet for tablet b7f3b609fb8042408ee785d19c6745a9 (DEFAULT_TABLE table=test-table [id=bc8fd25d2b7e43e88d0ed9aba264df6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:09:26.806617 15113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7f3b609fb8042408ee785d19c6745a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:09:26.816486 15325 tablet_bootstrap.cc:492] T b7f3b609fb8042408ee785d19c6745a9 P c92c5a7d5a7944329f0026645a4253bb: Bootstrap starting.
I20260926 03:09:26.820379 15325 tablet_bootstrap.cc:654] T b7f3b609fb8042408ee785d19c6745a9 P c92c5a7d5a7944329f0026645a4253bb: Neither blocks nor log segments found. Creating new log.
I20260926 03:09:26.821326 15326 tablet_bootstrap.cc:492] T b7f3b609fb8042408ee785d19c6745a9 P 5d61fa4921724c1abb3d38b989a0aafc: Bootstrap starting.
I20260926 03:09:26.822533 15327 tablet_bootstrap.cc:492] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63: Bootstrap starting.
I20260926 03:09:26.822726 15325 log.cc:824] T b7f3b609fb8042408ee785d19c6745a9 P c92c5a7d5a7944329f0026645a4253bb: Log is configured to *not* fsync() on all Append() calls
I20260926 03:09:26.825836 15326 tablet_bootstrap.cc:654] T b7f3b609fb8042408ee785d19c6745a9 P 5d61fa4921724c1abb3d38b989a0aafc: Neither blocks nor log segments found. Creating new log.
I20260926 03:09:26.825860 15327 tablet_bootstrap.cc:654] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63: Neither blocks nor log segments found. Creating new log.
I20260926 03:09:26.826305 15325 tablet_bootstrap.cc:492] T b7f3b609fb8042408ee785d19c6745a9 P c92c5a7d5a7944329f0026645a4253bb: No bootstrap required, opened a new log
I20260926 03:09:26.826596 15325 ts_tablet_manager.cc:1397] T b7f3b609fb8042408ee785d19c6745a9 P c92c5a7d5a7944329f0026645a4253bb: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260926 03:09:26.827135 15326 log.cc:824] T b7f3b609fb8042408ee785d19c6745a9 P 5d61fa4921724c1abb3d38b989a0aafc: Log is configured to *not* fsync() on all Append() calls
I20260926 03:09:26.827135 15327 log.cc:824] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63: Log is configured to *not* fsync() on all Append() calls
I20260926 03:09:26.833382 15327 tablet_bootstrap.cc:492] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63: No bootstrap required, opened a new log
I20260926 03:09:26.833534 15326 tablet_bootstrap.cc:492] T b7f3b609fb8042408ee785d19c6745a9 P 5d61fa4921724c1abb3d38b989a0aafc: No bootstrap required, opened a new log
I20260926 03:09:26.833709 15327 ts_tablet_manager.cc:1397] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260926 03:09:26.833828 15326 ts_tablet_manager.cc:1397] T b7f3b609fb8042408ee785d19c6745a9 P 5d61fa4921724c1abb3d38b989a0aafc: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260926 03:09:26.835681 15325 raft_consensus.cc:348] T b7f3b609fb8042408ee785d19c6745a9 P c92c5a7d5a7944329f0026645a4253bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92c5a7d5a7944329f0026645a4253bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43181 } } peers { permanent_uuid: "5d61fa4921724c1abb3d38b989a0aafc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45459 } } peers { permanent_uuid: "8ef908b570fb41e78d622d1b09d75a63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36847 } }
I20260926 03:09:26.836163 15325 raft_consensus.cc:374] T b7f3b609fb8042408ee785d19c6745a9 P c92c5a7d5a7944329f0026645a4253bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:09:26.836256 15325 raft_consensus.cc:729] T b7f3b609fb8042408ee785d19c6745a9 P c92c5a7d5a7944329f0026645a4253bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92c5a7d5a7944329f0026645a4253bb, State: Initialized, Role: FOLLOWER
I20260926 03:09:26.837033 15325 consensus_queue.cc:260] T b7f3b609fb8042408ee785d19c6745a9 P c92c5a7d5a7944329f0026645a4253bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92c5a7d5a7944329f0026645a4253bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43181 } } peers { permanent_uuid: "5d61fa4921724c1abb3d38b989a0aafc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45459 } } peers { permanent_uuid: "8ef908b570fb41e78d622d1b09d75a63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36847 } }
I20260926 03:09:26.838239 15309 heartbeater.cc:503] Master 127.0.97.62:40383 was elected leader, sending a full tablet report...
I20260926 03:09:26.839037 15325 ts_tablet_manager.cc:1428] T b7f3b609fb8042408ee785d19c6745a9 P c92c5a7d5a7944329f0026645a4253bb: Time spent starting tablet: real 0.012s user 0.003s sys 0.009s
I20260926 03:09:26.842113 15046 heartbeater.cc:503] Master 127.0.97.62:40383 was elected leader, sending a full tablet report...
I20260926 03:09:26.843431 15326 raft_consensus.cc:348] T b7f3b609fb8042408ee785d19c6745a9 P 5d61fa4921724c1abb3d38b989a0aafc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92c5a7d5a7944329f0026645a4253bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43181 } } peers { permanent_uuid: "5d61fa4921724c1abb3d38b989a0aafc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45459 } } peers { permanent_uuid: "8ef908b570fb41e78d622d1b09d75a63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36847 } }
I20260926 03:09:26.843935 15326 raft_consensus.cc:374] T b7f3b609fb8042408ee785d19c6745a9 P 5d61fa4921724c1abb3d38b989a0aafc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:09:26.844040 15326 raft_consensus.cc:729] T b7f3b609fb8042408ee785d19c6745a9 P 5d61fa4921724c1abb3d38b989a0aafc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d61fa4921724c1abb3d38b989a0aafc, State: Initialized, Role: FOLLOWER
I20260926 03:09:26.844702 15326 consensus_queue.cc:260] T b7f3b609fb8042408ee785d19c6745a9 P 5d61fa4921724c1abb3d38b989a0aafc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92c5a7d5a7944329f0026645a4253bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43181 } } peers { permanent_uuid: "5d61fa4921724c1abb3d38b989a0aafc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45459 } } peers { permanent_uuid: "8ef908b570fb41e78d622d1b09d75a63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36847 } }
I20260926 03:09:26.844785 15327 raft_consensus.cc:348] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92c5a7d5a7944329f0026645a4253bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43181 } } peers { permanent_uuid: "5d61fa4921724c1abb3d38b989a0aafc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45459 } } peers { permanent_uuid: "8ef908b570fb41e78d622d1b09d75a63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36847 } }
I20260926 03:09:26.845263 15327 raft_consensus.cc:374] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:09:26.845371 15327 raft_consensus.cc:729] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ef908b570fb41e78d622d1b09d75a63, State: Initialized, Role: FOLLOWER
I20260926 03:09:26.846108 15327 consensus_queue.cc:260] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92c5a7d5a7944329f0026645a4253bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43181 } } peers { permanent_uuid: "5d61fa4921724c1abb3d38b989a0aafc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45459 } } peers { permanent_uuid: "8ef908b570fb41e78d622d1b09d75a63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36847 } }
I20260926 03:09:26.847285 15326 ts_tablet_manager.cc:1428] T b7f3b609fb8042408ee785d19c6745a9 P 5d61fa4921724c1abb3d38b989a0aafc: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260926 03:09:26.848282 15327 ts_tablet_manager.cc:1428] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260926 03:09:26.848697 15178 heartbeater.cc:503] Master 127.0.97.62:40383 was elected leader, sending a full tablet report...
I20260926 03:09:26.933650 15333 raft_consensus.cc:482] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:09:26.933969 15333 raft_consensus.cc:504] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92c5a7d5a7944329f0026645a4253bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43181 } } peers { permanent_uuid: "5d61fa4921724c1abb3d38b989a0aafc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45459 } } peers { permanent_uuid: "8ef908b570fb41e78d622d1b09d75a63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36847 } }
I20260926 03:09:26.935832 15333 leader_election.cc:288] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c92c5a7d5a7944329f0026645a4253bb (127.0.97.3:43181), 5d61fa4921724c1abb3d38b989a0aafc (127.0.97.1:45459)
I20260926 03:09:26.943365 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:26.936527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52871 (local address 127.0.97.1:45459)
0926 03:09:26.936778 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:26.936784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:26.937426 (+ 642us) server_negotiation.cc:407] Connection header received
0926 03:09:26.937575 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:26.937579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:26.937638 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:26.937709 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:09:26.938658 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.939596 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.941612 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.942395 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.942547 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:26.942650 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:26.942989 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 03:09:26.943114 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 03:09:26.943375 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:26.936554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54125 (local address 127.0.97.3:43181)
0926 03:09:26.936766 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:26.936773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:26.937210 (+ 437us) server_negotiation.cc:407] Connection header received
0926 03:09:26.937412 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:26.937416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:26.937478 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:26.937551 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:09:26.938474 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.939354 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.941614 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.942411 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.942552 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:26.942659 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:26.943002 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 03:09:26.943135 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:09:26.943532 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:26.936284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43181 (local address 127.0.97.2:54125)
0926 03:09:26.937086 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:26.937105 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:26.937277 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:26.937586 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:26.937591 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:26.937623 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:09:26.938186 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:26.938201 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.939522 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.939530 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:26.941456 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:26.941470 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.942689 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.942696 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:26.942833 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:26.942879 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 03:09:26.942955 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 03:09:26.943110 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":166,"thread_start_us":479,"threads_started":1}
I20260926 03:09:26.943904 15335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:26.936368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45459 (local address 127.0.97.2:52871)
0926 03:09:26.937296 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:26.937310 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:26.937442 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:26.937765 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:26.937769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:26.937800 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:09:26.938433 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:26.938446 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.939739 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.939744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:26.941459 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:26.941471 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.942655 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.942663 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:26.942822 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:26.942861 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 03:09:26.942942 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 03:09:26.943095 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":49,"thread_start_us":579,"threads_started":1}
I20260926 03:09:26.944507 15264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7f3b609fb8042408ee785d19c6745a9" candidate_uuid: "8ef908b570fb41e78d622d1b09d75a63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92c5a7d5a7944329f0026645a4253bb" is_pre_election: true
I20260926 03:09:26.944859 15001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7f3b609fb8042408ee785d19c6745a9" candidate_uuid: "8ef908b570fb41e78d622d1b09d75a63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d61fa4921724c1abb3d38b989a0aafc" is_pre_election: true
I20260926 03:09:26.945099 15264 raft_consensus.cc:2454] T b7f3b609fb8042408ee785d19c6745a9 P c92c5a7d5a7944329f0026645a4253bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ef908b570fb41e78d622d1b09d75a63 in term 0.
I20260926 03:09:26.945372 15001 raft_consensus.cc:2454] T b7f3b609fb8042408ee785d19c6745a9 P 5d61fa4921724c1abb3d38b989a0aafc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ef908b570fb41e78d622d1b09d75a63 in term 0.
I20260926 03:09:26.945959 15068 leader_election.cc:302] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ef908b570fb41e78d622d1b09d75a63, c92c5a7d5a7944329f0026645a4253bb; no voters:
I20260926 03:09:26.946376 15333 raft_consensus.cc:2781] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:09:26.946527 15333 raft_consensus.cc:482] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:09:26.946627 15333 raft_consensus.cc:3037] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:09:26.948896 15333 raft_consensus.cc:504] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92c5a7d5a7944329f0026645a4253bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43181 } } peers { permanent_uuid: "5d61fa4921724c1abb3d38b989a0aafc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45459 } } peers { permanent_uuid: "8ef908b570fb41e78d622d1b09d75a63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36847 } }
I20260926 03:09:26.949522 15333 leader_election.cc:288] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63 [CANDIDATE]: Term 1 election: Requested vote from peers c92c5a7d5a7944329f0026645a4253bb (127.0.97.3:43181), 5d61fa4921724c1abb3d38b989a0aafc (127.0.97.1:45459)
I20260926 03:09:26.950095 15264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7f3b609fb8042408ee785d19c6745a9" candidate_uuid: "8ef908b570fb41e78d622d1b09d75a63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92c5a7d5a7944329f0026645a4253bb"
I20260926 03:09:26.950090 15001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7f3b609fb8042408ee785d19c6745a9" candidate_uuid: "8ef908b570fb41e78d622d1b09d75a63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d61fa4921724c1abb3d38b989a0aafc"
I20260926 03:09:26.950284 15264 raft_consensus.cc:3037] T b7f3b609fb8042408ee785d19c6745a9 P c92c5a7d5a7944329f0026645a4253bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:09:26.950294 15001 raft_consensus.cc:3037] T b7f3b609fb8042408ee785d19c6745a9 P 5d61fa4921724c1abb3d38b989a0aafc [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:09:26.952781 15001 raft_consensus.cc:2454] T b7f3b609fb8042408ee785d19c6745a9 P 5d61fa4921724c1abb3d38b989a0aafc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ef908b570fb41e78d622d1b09d75a63 in term 1.
I20260926 03:09:26.952759 15264 raft_consensus.cc:2454] T b7f3b609fb8042408ee785d19c6745a9 P c92c5a7d5a7944329f0026645a4253bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ef908b570fb41e78d622d1b09d75a63 in term 1.
I20260926 03:09:26.953375 15068 leader_election.cc:302] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5d61fa4921724c1abb3d38b989a0aafc, 8ef908b570fb41e78d622d1b09d75a63, c92c5a7d5a7944329f0026645a4253bb; no voters:
I20260926 03:09:26.953744 15333 raft_consensus.cc:2781] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:09:26.954209 15333 raft_consensus.cc:686] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63 [term 1 LEADER]: Becoming Leader. State: Replica: 8ef908b570fb41e78d622d1b09d75a63, State: Running, Role: LEADER
I20260926 03:09:26.954734 15333 consensus_queue.cc:237] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92c5a7d5a7944329f0026645a4253bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43181 } } peers { permanent_uuid: "5d61fa4921724c1abb3d38b989a0aafc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45459 } } peers { permanent_uuid: "8ef908b570fb41e78d622d1b09d75a63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36847 } }
I20260926 03:09:26.960031 14861 catalog_manager.cc:5495] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ef908b570fb41e78d622d1b09d75a63 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "8ef908b570fb41e78d622d1b09d75a63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92c5a7d5a7944329f0026645a4253bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d61fa4921724c1abb3d38b989a0aafc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ef908b570fb41e78d622d1b09d75a63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36847 } health_report { overall_health: HEALTHY } } }
W20260926 03:09:26.963971 15310 tablet.cc:2369] T b7f3b609fb8042408ee785d19c6745a9 P c92c5a7d5a7944329f0026645a4253bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:09:26.991617 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:26.985995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53140 (local address 127.0.97.2:36847)
0926 03:09:26.986263 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:26.986269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:26.986289 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:09:26.986466 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:26.986470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:26.986596 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:26.986740 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:09:26.987116 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.987885 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.989379 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:26.989799 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:26.989907 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:26.990440 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:09:26.990999 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:09:26.991141 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:26.991240 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 03:09:26.991329 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 03:09:27.003075 15001 raft_consensus.cc:1261] T b7f3b609fb8042408ee785d19c6745a9 P 5d61fa4921724c1abb3d38b989a0aafc [term 1 FOLLOWER]: Refusing update from remote peer 8ef908b570fb41e78d622d1b09d75a63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:09:27.003075 15264 raft_consensus.cc:1261] T b7f3b609fb8042408ee785d19c6745a9 P c92c5a7d5a7944329f0026645a4253bb [term 1 FOLLOWER]: Refusing update from remote peer 8ef908b570fb41e78d622d1b09d75a63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:09:27.004684 15336 consensus_queue.cc:1035] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92c5a7d5a7944329f0026645a4253bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:09:27.005144 15333 consensus_queue.cc:1035] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d61fa4921724c1abb3d38b989a0aafc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 03:09:27.015805 15179 tablet.cc:2369] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:09:27.019776 15340 mvcc.cc:205] Tried to move back new op lower bound from 7333446316032516096 to 7333446315850842112. Current Snapshot: MvccSnapshot[applied={T|T < 7333446316032516096}]
I20260926 03:09:27.022293 15339 mvcc.cc:205] Tried to move back new op lower bound from 7333446316032516096 to 7333446315850842112. Current Snapshot: MvccSnapshot[applied={T|T < 7333446316032516096}]
I20260926 03:09:27.037093 14860 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:09:27.037463 14860 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:09:27.042030 14860 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933: Sending DeleteTablet for 3 replicas of tablet b7f3b609fb8042408ee785d19c6745a9
I20260926 03:09:27.043552 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 14898
I20260926 03:09:27.043548 15244 tablet_service.cc:1514] Processing DeleteTablet for tablet b7f3b609fb8042408ee785d19c6745a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:09:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55994
I20260926 03:09:27.044379 15113 tablet_service.cc:1514] Processing DeleteTablet for tablet b7f3b609fb8042408ee785d19c6745a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:09:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53128
I20260926 03:09:27.044552 15349 tablet_replica.cc:330] T b7f3b609fb8042408ee785d19c6745a9 P c92c5a7d5a7944329f0026645a4253bb: stopping tablet replica
I20260926 03:09:27.045015 15349 raft_consensus.cc:2229] T b7f3b609fb8042408ee785d19c6745a9 P c92c5a7d5a7944329f0026645a4253bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:09:27.045400 15350 tablet_replica.cc:330] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63: stopping tablet replica
I20260926 03:09:27.045569 15349 raft_consensus.cc:2258] T b7f3b609fb8042408ee785d19c6745a9 P c92c5a7d5a7944329f0026645a4253bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:09:27.045804 14981 tablet_service.cc:1514] Processing DeleteTablet for tablet b7f3b609fb8042408ee785d19c6745a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:09:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59342
I20260926 03:09:27.045940 15350 raft_consensus.cc:2229] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:09:27.046746 15351 tablet_replica.cc:330] T b7f3b609fb8042408ee785d19c6745a9 P 5d61fa4921724c1abb3d38b989a0aafc: stopping tablet replica
I20260926 03:09:27.046809 15350 raft_consensus.cc:2258] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:09:27.047183 15351 raft_consensus.cc:2229] T b7f3b609fb8042408ee785d19c6745a9 P 5d61fa4921724c1abb3d38b989a0aafc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:09:27.048036 15351 raft_consensus.cc:2258] T b7f3b609fb8042408ee785d19c6745a9 P 5d61fa4921724c1abb3d38b989a0aafc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:09:27.049216 15350 ts_tablet_manager.cc:1905] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:09:27.050158 15351 ts_tablet_manager.cc:1905] T b7f3b609fb8042408ee785d19c6745a9 P 5d61fa4921724c1abb3d38b989a0aafc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:09:27.050400 15349 ts_tablet_manager.cc:1905] T b7f3b609fb8042408ee785d19c6745a9 P c92c5a7d5a7944329f0026645a4253bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:09:27.057046 15350 ts_tablet_manager.cc:1918] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:09:27.057049 15351 ts_tablet_manager.cc:1918] T b7f3b609fb8042408ee785d19c6745a9 P 5d61fa4921724c1abb3d38b989a0aafc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:09:27.057237 15351 log.cc:1196] T b7f3b609fb8042408ee785d19c6745a9 P 5d61fa4921724c1abb3d38b989a0aafc: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/wal/wals/b7f3b609fb8042408ee785d19c6745a9
I20260926 03:09:27.057411 15350 log.cc:1196] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/wal/wals/b7f3b609fb8042408ee785d19c6745a9
I20260926 03:09:27.057914 15351 ts_tablet_manager.cc:1939] T b7f3b609fb8042408ee785d19c6745a9 P 5d61fa4921724c1abb3d38b989a0aafc: Deleting consensus metadata
I20260926 03:09:27.058033 15350 ts_tablet_manager.cc:1939] T b7f3b609fb8042408ee785d19c6745a9 P 8ef908b570fb41e78d622d1b09d75a63: Deleting consensus metadata
I20260926 03:09:27.060523 14846 catalog_manager.cc:4841] TS 8ef908b570fb41e78d622d1b09d75a63 (127.0.97.2:36847): tablet b7f3b609fb8042408ee785d19c6745a9 (table test-table [id=bc8fd25d2b7e43e88d0ed9aba264df6b]) successfully deleted
I20260926 03:09:27.060883 14846 catalog_manager.cc:4841] TS 5d61fa4921724c1abb3d38b989a0aafc (127.0.97.1:45459): tablet b7f3b609fb8042408ee785d19c6745a9 (table test-table [id=bc8fd25d2b7e43e88d0ed9aba264df6b]) successfully deleted
I20260926 03:09:27.064085 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:27.046838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59354 (local address 127.0.97.1:45459)
0926 03:09:27.046996 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:27.047001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:27.047023 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:09:27.047085 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:27.047090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:27.047203 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:27.047336 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:27.049211 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:27.050369 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:27.052315 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:27.052928 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:27.053063 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:27.056582 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:27.056602 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:27.056621 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:27.056667 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:27.060835 (+ 4168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:27.061667 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:27.061677 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:27.061681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:27.061859 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:27.062394 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:27.062399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:27.062403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:27.062889 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:27.063277 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:27.063489 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 03:09:27.063728 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 03:09:27.064837 15041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:09:27.065529 15349 ts_tablet_manager.cc:1918] T b7f3b609fb8042408ee785d19c6745a9 P c92c5a7d5a7944329f0026645a4253bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:09:27.065752 15349 log.cc:1196] T b7f3b609fb8042408ee785d19c6745a9 P c92c5a7d5a7944329f0026645a4253bb: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/wal/wals/b7f3b609fb8042408ee785d19c6745a9
I20260926 03:09:27.066635 15349 ts_tablet_manager.cc:1939] T b7f3b609fb8042408ee785d19c6745a9 P c92c5a7d5a7944329f0026645a4253bb: Deleting consensus metadata
I20260926 03:09:27.069463 14847 catalog_manager.cc:4841] TS c92c5a7d5a7944329f0026645a4253bb (127.0.97.3:43181): tablet b7f3b609fb8042408ee785d19c6745a9 (table test-table [id=bc8fd25d2b7e43e88d0ed9aba264df6b]) successfully deleted
I20260926 03:09:27.155360 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 14898
I20260926 03:09:27.178695 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 15050
I20260926 03:09:27.195427 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:27.179524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53154 (local address 127.0.97.2:36847)
0926 03:09:27.179708 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:27.179713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:27.179914 (+ 201us) server_negotiation.cc:407] Connection header received
0926 03:09:27.180073 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:27.180079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:27.180181 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:27.180300 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:27.181626 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:27.183099 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:27.184514 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:27.185169 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:27.185309 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:27.187945 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:27.187963 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:27.187984 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:27.188040 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:27.192101 (+ 4061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:27.193055 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:27.193065 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:27.193069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:27.193253 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:27.193618 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:27.193623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:27.193627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:27.194144 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:27.194486 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:27.194771 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 03:09:27.195012 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 03:09:27.196223 15173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:09:27.309732 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 15050
I20260926 03:09:27.328320 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 15181
I20260926 03:09:27.341984 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:27.329150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56000 (local address 127.0.97.3:43181)
0926 03:09:27.329361 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:27.329367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:27.329390 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:09:27.329585 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:27.329589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:27.329655 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:27.329737 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:27.331054 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:27.331763 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:27.333044 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:27.333445 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:27.333554 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:27.336235 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:27.336250 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:27.336267 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:27.336306 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:27.339058 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:27.339844 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:27.339853 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:27.339855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:27.339993 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:27.340375 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:27.340380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:27.340382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:27.340790 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:27.341079 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:27.341386 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 03:09:27.341610 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 03:09:27.342718 15304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:09:27.433288 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 15181
I20260926 03:09:27.446544 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 14828
I20260926 03:09:27.447829 14891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:09:27.560171 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 14828
I20260926 03:09:27.578404 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40383
--webserver_interface=127.0.97.62
--webserver_port=42561
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:09:27.911258 15356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:09:27.911556 15356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:09:27.911645 15356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:09:27.918937 15356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:09:27.919047 15356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:09:27.919106 15356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:09:27.919173 15356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:09:27.919216 15356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:09:27.929163 15356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=42561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:09:27.930471 15356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:09:27.931842 15356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:09:27.943320 15361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:27.943233 15362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:27.943282 15364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:27.943591 15356 server_base.cc:1034] running on GCE node
I20260926 03:09:27.944619 15356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:09:27.946039 15356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:09:27.947289 15356 hybrid_clock.cc:648] HybridClock initialized: now 1790392167947222 us; error 53 us; skew 500 ppm
I20260926 03:09:27.947659 15356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:09:27 dist-test-slave-hmbw krb5kdc[14277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:09:27.950896 15356 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:09:27.952535 15356 webserver.cc:460] Webserver started at http://127.0.97.62:42561/ using document root <none> and password file <none>
I20260926 03:09:27.953217 15356 fs_manager.cc:359] Metadata directory not provided
I20260926 03:09:27.953346 15356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:09:27.959506 15356 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 03:09:27.962791 15371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:27.964241 15356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:09:27.964433 15356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "95a5c9e578354ddbabdef1290ed59933"
format_stamp: "Formatted at 2026-09-26 03:09:22 on dist-test-slave-hmbw"
server_key: "3dc1381332c74c6378afc8bd10529a58"
server_key_iv: "59ff032508fc965e3ef7d7740ddf0528"
server_key_version: "encryptionkey@0"
I20260926 03:09:27.965547 15356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:09:27.995573 15356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:09:28.000066 15356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:09:28.000363 15356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:09:28.019526 15356 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40383
I20260926 03:09:28.019572 15422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40383 every 8 connection(s)
I20260926 03:09:28.021129 15356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:09:28.028172 15423 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:09:28.030645 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 15356
I20260926 03:09:28.031419 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45459
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:40383
--webserver_port=37257
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:09:28.031719 15423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933: Bootstrap starting.
I20260926 03:09:28.052347 15423 log.cc:824] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933: Log is configured to *not* fsync() on all Append() calls
I20260926 03:09:28.069725 15423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:09:28.070264 15423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933: Bootstrap complete.
I20260926 03:09:28.079475 15423 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a5c9e578354ddbabdef1290ed59933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } }
I20260926 03:09:28.080391 15423 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a5c9e578354ddbabdef1290ed59933, State: Initialized, Role: FOLLOWER
I20260926 03:09:28.081002 15423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a5c9e578354ddbabdef1290ed59933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } }
I20260926 03:09:28.081229 15423 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:09:28.081342 15423 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:09:28.081496 15423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [term 2 FOLLOWER]: Advancing to term 3
I20260926 03:09:28.085589 15423 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a5c9e578354ddbabdef1290ed59933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } }
I20260926 03:09:28.086108 15423 leader_election.cc:302] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95a5c9e578354ddbabdef1290ed59933; no voters:
I20260926 03:09:28.086781 15423 leader_election.cc:288] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 03:09:28.086938 15427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 03:09:28.088020 15427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [term 3 LEADER]: Becoming Leader. State: Replica: 95a5c9e578354ddbabdef1290ed59933, State: Running, Role: LEADER
I20260926 03:09:28.088608 15427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a5c9e578354ddbabdef1290ed59933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } }
I20260926 03:09:28.088989 15423 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:09:28.091090 15429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95a5c9e578354ddbabdef1290ed59933. Latest consensus state: current_term: 3 leader_uuid: "95a5c9e578354ddbabdef1290ed59933" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a5c9e578354ddbabdef1290ed59933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } } }
I20260926 03:09:28.091466 15429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [sys.catalog]: This master's current role is: LEADER
I20260926 03:09:28.092002 15428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "95a5c9e578354ddbabdef1290ed59933" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a5c9e578354ddbabdef1290ed59933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } } }
I20260926 03:09:28.092260 15428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95a5c9e578354ddbabdef1290ed59933 [sys.catalog]: This master's current role is: LEADER
I20260926 03:09:28.102869 15433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:09:28.112548 15433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:09:28.114321 15433 catalog_manager.cc:1259] Loaded cluster ID: ad5e00c37c6d46b0abcd168ed408069d
I20260926 03:09:28.114454 15433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 03:09:28.116897 15433 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f37e65979ad google::LogMessage::Fail() at ??:0
@ 0x7f37e659a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f37e65974de google::LogMessage::Flush() at ??:0
@ 0x7f37e6598a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f37fbfca512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
I20260926 03:09:28.139276 15445 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f37fbfc7976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f37fc04f610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f37fc04f439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f37e91e3345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f37e91e691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f37e91e6799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f37e91c1eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f37ec49b6db start_thread at ??:0
@ 0x7f37e543161f clone at ??:0
W20260926 03:09:28.393316 15425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:09:28.393601 15425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:09:28.393788 15425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:09:28.401218 15425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:09:28.401355 15425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:09:28.401515 15425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:09:28.411448 15425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45459
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=37257
--tserver_master_addrs=127.0.97.62:40383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:09:28.412768 15425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:09:28.414098 15425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:09:28.427095 15464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:28.427054 15467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:28.427117 15465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:28.428094 15425 server_base.cc:1034] running on GCE node
I20260926 03:09:28.428789 15425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:09:28.429761 15425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:09:28.431034 15425 hybrid_clock.cc:648] HybridClock initialized: now 1790392168430970 us; error 40 us; skew 500 ppm
I20260926 03:09:28.431427 15425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:09:28 dist-test-slave-hmbw krb5kdc[14277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:09:28.434757 15425 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:09:28.436257 15425 webserver.cc:460] Webserver started at http://127.0.97.1:37257/ using document root <none> and password file <none>
I20260926 03:09:28.436964 15425 fs_manager.cc:359] Metadata directory not provided
I20260926 03:09:28.437083 15425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:09:28.442688 15425 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:09:28.445572 15474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:28.447046 15425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:09:28.447222 15425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "5d61fa4921724c1abb3d38b989a0aafc"
format_stamp: "Formatted at 2026-09-26 03:09:22 on dist-test-slave-hmbw"
server_key: "b6daab19fae6d01d209882cd78ae2bf4"
server_key_iv: "d008da5ed934ac711aa4ffc6b15d38f9"
server_key_version: "encryptionkey@0"
I20260926 03:09:28.448331 15425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:09:28.468010 15425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:09:28.472569 15425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:09:28.472836 15425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:09:28.473858 15425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:09:28.475360 15425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:09:28.475463 15425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:28.475579 15425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:09:28.475646 15425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:28.506369 15425 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45459
I20260926 03:09:28.506474 15586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45459 every 8 connection(s)
I20260926 03:09:28.508917 15425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:09:28.513784 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 15425
I20260926 03:09:28.514376 15589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:28.512148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.1:34447)
0926 03:09:28.512930 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:28.513142 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":295,"threads_started":1}
I20260926 03:09:28.514546 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:36847
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:40383
--webserver_port=36537
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:09:28.515223 15587 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40383 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:40383: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
I20260926 03:09:28.516185 15589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:28.515795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.1:41427)
0926 03:09:28.515961 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:28.516049 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 03:09:28.517095 15589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:28.516730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.1:46015)
0926 03:09:28.516872 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:28.516964 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 03:09:28.517370 15587 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 03:09:28.845395 15590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:09:28.845682 15590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:09:28.845782 15590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:09:28.853277 15590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:09:28.853410 15590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:09:28.853535 15590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:09:28.863894 15590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:36847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36537
--tserver_master_addrs=127.0.97.62:40383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:09:28.865289 15590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:09:28.866655 15590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:09:28.879837 15595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:28.879992 15596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:28.880025 15598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:28.881382 15590 server_base.cc:1034] running on GCE node
I20260926 03:09:28.882016 15590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:09:28.883064 15590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:09:28.884267 15590 hybrid_clock.cc:648] HybridClock initialized: now 1790392168884206 us; error 51 us; skew 500 ppm
I20260926 03:09:28.884567 15590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:09:28 dist-test-slave-hmbw krb5kdc[14277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:09:28.888058 15590 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:09:28.889737 15590 webserver.cc:460] Webserver started at http://127.0.97.2:36537/ using document root <none> and password file <none>
I20260926 03:09:28.890455 15590 fs_manager.cc:359] Metadata directory not provided
I20260926 03:09:28.890607 15590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:09:28.896387 15590 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 03:09:28.899519 15605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:28.900983 15590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:09:28.901147 15590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "8ef908b570fb41e78d622d1b09d75a63"
format_stamp: "Formatted at 2026-09-26 03:09:23 on dist-test-slave-hmbw"
server_key: "8785fbe15b78d428044c1f521acdf7ee"
server_key_iv: "c6f284986fb496dddf33ef438350177e"
server_key_version: "encryptionkey@0"
I20260926 03:09:28.902209 15590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:09:28.914464 15590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:09:28.919211 15590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:09:28.919480 15590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:09:28.920493 15590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:09:28.922081 15590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:09:28.922186 15590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:28.922308 15590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:09:28.922375 15590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:28.954238 15590 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36847
I20260926 03:09:28.954313 15717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36847 every 8 connection(s)
I20260926 03:09:28.955787 15590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:09:28.957543 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 15590
I20260926 03:09:28.958230 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:43181
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:40383
--webserver_port=44513
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:09:28.960984 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:28.959111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.2:49229)
0926 03:09:28.959901 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:28.960082 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":304,"threads_started":1}
W20260926 03:09:28.961745 15718 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40383 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:40383: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
I20260926 03:09:28.962707 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:28.962323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.2:52365)
0926 03:09:28.962482 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:28.962589 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 03:09:28.963572 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:28.963258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.2:46657)
0926 03:09:28.963386 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:28.963462 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 03:09:28.963814 15718 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 03:09:29.293710 15721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:09:29.294019 15721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:09:29.294116 15721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:09:29.302021 15721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:09:29.302148 15721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:09:29.302275 15721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:09:29.312606 15721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:43181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=44513
--tserver_master_addrs=127.0.97.62:40383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:09:29.313915 15721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:09:29.315291 15721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:09:29.330835 15726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:29.332376 15727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:29.333092 15729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:29.333945 15721 server_base.cc:1034] running on GCE node
I20260926 03:09:29.334657 15721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:09:29.335709 15721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:09:29.337014 15721 hybrid_clock.cc:648] HybridClock initialized: now 1790392169336950 us; error 58 us; skew 500 ppm
I20260926 03:09:29.337333 15721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:09:29 dist-test-slave-hmbw krb5kdc[14277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:09:29.340670 15721 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:09:29.342206 15721 webserver.cc:460] Webserver started at http://127.0.97.3:44513/ using document root <none> and password file <none>
I20260926 03:09:29.343026 15721 fs_manager.cc:359] Metadata directory not provided
I20260926 03:09:29.343139 15721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:09:29.348929 15721 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 03:09:29.357992 15736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:29.359843 15721 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.004s sys 0.000s
I20260926 03:09:29.360028 15721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "c92c5a7d5a7944329f0026645a4253bb"
format_stamp: "Formatted at 2026-09-26 03:09:23 on dist-test-slave-hmbw"
server_key: "23f80321cad9451b9b6eef0cdaa13f5e"
server_key_iv: "bf4fd59836a5ce9055317d5d11bcbb5f"
server_key_version: "encryptionkey@0"
I20260926 03:09:29.361091 15721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:09:29.379861 15721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:09:29.384207 15721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:09:29.384462 15721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:09:29.385475 15721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:09:29.386878 15721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:09:29.386974 15721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:29.387080 15721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:09:29.387145 15721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:29.418236 15721 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43181
I20260926 03:09:29.418296 15848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43181 every 8 connection(s)
I20260926 03:09:29.419744 15721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:09:29.422260 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 15721
I20260926 03:09:29.424916 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:29.422694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.3:46979)
0926 03:09:29.423675 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:29.423946 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":411,"threads_started":1}
W20260926 03:09:29.425657 15849 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40383 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:40383: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
I20260926 03:09:29.426442 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:29.426116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.3:35745)
0926 03:09:29.426271 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:29.426336 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 03:09:29.427369 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:29.427015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.3:54607)
0926 03:09:29.427162 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:29.427258 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 03:09:29.427610 15849 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 03:09:29.518887 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:29.518030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.1:46247)
0926 03:09:29.518533 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:29.518653 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":228,"threads_started":1}
I20260926 03:09:29.965413 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:29.964470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.2:49657)
0926 03:09:29.965041 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:29.965153 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":273,"threads_started":1}
I20260926 03:09:30.429128 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:30.428263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.3:49433)
0926 03:09:30.428796 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:30.428893 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":251,"threads_started":1}
I20260926 03:09:30.520756 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:30.519912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.1:57057)
0926 03:09:30.520475 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:30.520581 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":270,"threads_started":1}
I20260926 03:09:30.967248 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:30.966407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.2:49649)
0926 03:09:30.966974 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:30.967068 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":296,"threads_started":1}
I20260926 03:09:31.431030 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:31.430189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.3:40841)
0926 03:09:31.430749 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:31.430855 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":255,"threads_started":1}
I20260926 03:09:31.522768 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:31.521901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.1:58223)
0926 03:09:31.522426 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:31.522564 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":301,"threads_started":1}
I20260926 03:09:31.968948 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:31.968229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.2:48335)
0926 03:09:31.968688 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:31.968776 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":226,"threads_started":1}
I20260926 03:09:32.432878 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:32.432101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.3:35057)
0926 03:09:32.432596 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:32.432686 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":276,"threads_started":1}
I20260926 03:09:32.524612 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:32.523749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.1:37685)
0926 03:09:32.524306 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:32.524403 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":245,"threads_started":1}
I20260926 03:09:32.970890 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:32.970016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.2:44089)
0926 03:09:32.970607 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:32.970690 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":332,"threads_started":1}
I20260926 03:09:33.434473 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:33.433750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.3:41407)
0926 03:09:33.434208 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:33.434300 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":243,"threads_started":1}
I20260926 03:09:33.526432 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:33.525636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.1:45753)
0926 03:09:33.526151 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:33.526240 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":260,"threads_started":1}
I20260926 03:09:33.972779 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:33.971932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.2:40757)
0926 03:09:33.972461 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:33.972563 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":259,"threads_started":1}
I20260926 03:09:34.436246 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:34.435472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.3:46845)
0926 03:09:34.435940 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:34.436018 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":254,"threads_started":1}
I20260926 03:09:34.528251 15868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:34.527430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.1:45967)
0926 03:09:34.527935 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:34.528032 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":269,"threads_started":1}
I20260926 03:09:34.974709 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:34.973748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.2:47161)
0926 03:09:34.974321 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:34.974443 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":252,"threads_started":1}
I20260926 03:09:35.438007 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:35.437239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.3:44733)
0926 03:09:35.437760 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:35.437847 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":254,"threads_started":1}
I20260926 03:09:35.530063 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:35.529253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.1:55837)
0926 03:09:35.529750 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:35.529836 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":219,"threads_started":1}
I20260926 03:09:35.976629 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:35.975704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.2:59265)
0926 03:09:35.976319 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:35.976400 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":328,"threads_started":1}
I20260926 03:09:36.439795 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:36.439010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.3:44995)
0926 03:09:36.439481 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:36.439562 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":209,"threads_started":1}
I20260926 03:09:36.531857 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:36.531099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.1:60185)
0926 03:09:36.531597 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:36.531684 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":245,"threads_started":1}
I20260926 03:09:36.978686 15875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:36.977755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.2:42803)
0926 03:09:36.978370 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:36.978470 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":249,"threads_started":1}
I20260926 03:09:37.441618 15876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:37.440806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.3:48135)
0926 03:09:37.441346 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:37.441448 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":270,"threads_started":1}
I20260926 03:09:37.533774 15877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:37.532920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.1:33009)
0926 03:09:37.533510 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:37.533593 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":326,"threads_started":1}
I20260926 03:09:37.980652 15878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:37.979840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.2:43733)
0926 03:09:37.980372 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:37.980463 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":285,"threads_started":1}
I20260926 03:09:38.443465 15879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:38.442708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.3:37185)
0926 03:09:38.443197 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:38.443284 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":232,"threads_started":1}
I20260926 03:09:38.535897 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:38.535065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.1:50933)
0926 03:09:38.535651 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:38.535730 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":314,"threads_started":1}
I20260926 03:09:38.982452 15881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:38.981692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.2:49175)
0926 03:09:38.982160 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:38.982286 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":243,"threads_started":1}
I20260926 03:09:39.445677 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:39.444449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.3:50805)
0926 03:09:39.445371 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:39.445455 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":816,"thread_start_us":698,"threads_started":1}
I20260926 03:09:39.537551 15883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:39.536836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.1:60661)
0926 03:09:39.537282 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:39.537371 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":218,"threads_started":1}
I20260926 03:09:39.984424 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:39.983574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.2:57685)
0926 03:09:39.984074 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:39.984237 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":229,"threads_started":1}
I20260926 03:09:40.447453 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:40.446695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.3:46881)
0926 03:09:40.447165 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:40.447269 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":231,"threads_started":1}
I20260926 03:09:40.539559 15886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:40.538780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.1:45297)
0926 03:09:40.539304 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:40.539393 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":289,"threads_started":1}
I20260926 03:09:40.986343 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:40.985550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.2:42765)
0926 03:09:40.986051 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:40.986156 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":213,"threads_started":1}
I20260926 03:09:41.449194 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:41.448475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.3:48867)
0926 03:09:41.448927 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:41.449017 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":224,"threads_started":1}
I20260926 03:09:41.541448 15889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:41.540604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.1:39371)
0926 03:09:41.541160 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:41.541255 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":279,"threads_started":1}
I20260926 03:09:41.988391 15890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:41.987512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.2:59471)
0926 03:09:41.988029 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:41.988196 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":239,"threads_started":1}
I20260926 03:09:42.451047 15891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:42.450194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.3:41107)
0926 03:09:42.450748 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:42.450842 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":239,"threads_started":1}
I20260926 03:09:42.543414 15892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:42.542563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.1:50659)
0926 03:09:42.543134 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:42.543226 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":247,"threads_started":1}
I20260926 03:09:42.990228 15893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:42.989399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.2:53775)
0926 03:09:42.989957 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:42.990044 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":256,"threads_started":1}
I20260926 03:09:43.452942 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:43.452180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.3:35925)
0926 03:09:43.452660 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:43.452751 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":242,"threads_started":1}
I20260926 03:09:43.545502 15895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:43.544711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.1:38987)
0926 03:09:43.545203 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:43.545299 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":243,"threads_started":1}
I20260926 03:09:43.992333 15896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:43.991627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.2:46115)
0926 03:09:43.992067 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:43.992164 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":217,"threads_started":1}
I20260926 03:09:44.430490 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 15425
I20260926 03:09:44.449784 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:44.431767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49802 (local address 127.0.97.1:45459)
0926 03:09:44.432383 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:44.432397 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:44.432434 (+ 37us) server_negotiation.cc:407] Connection header received
0926 03:09:44.434040 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:44.434050 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:44.434532 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:44.434838 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:44.436319 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:44.437588 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:44.438961 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:44.439744 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:44.439889 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:44.442823 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:44.442833 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:44.442852 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:44.442898 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:44.446168 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:44.447142 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:44.447150 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:44.447157 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:44.447349 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:44.447761 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:44.447764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:44.447766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:44.448423 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:44.448672 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:44.449099 (+ 427us) server_negotiation.cc:299] Negotiation successful
0926 03:09:44.449469 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":294,"threads_started":1}
I20260926 03:09:44.451329 15582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:09:44.455344 15899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:44.454363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40383 (local address 127.0.97.3:51731)
0926 03:09:44.454927 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:44.455104 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":281,"threads_started":1}
I20260926 03:09:44.536329 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 15425
I20260926 03:09:44.551072 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 15590
I20260926 03:09:44.569758 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:44.552040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44056 (local address 127.0.97.2:36847)
0926 03:09:44.552611 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:44.552625 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:44.552668 (+ 43us) server_negotiation.cc:407] Connection header received
0926 03:09:44.554107 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:44.554117 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:44.554656 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:44.555023 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:44.556451 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:44.557755 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:44.559186 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:44.559949 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:44.560156 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:44.562885 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:44.562894 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:44.562913 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:44.562960 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:44.566044 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:44.567023 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:44.567030 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:44.567038 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:44.567240 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:44.567639 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:44.567644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:44.567647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:44.568220 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:44.568526 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:44.568995 (+ 469us) server_negotiation.cc:299] Negotiation successful
0926 03:09:44.569459 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":252,"threads_started":1}
I20260926 03:09:44.571352 15713 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:09:44.654841 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 15590
I20260926 03:09:44.669469 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 15721
I20260926 03:09:44.687978 15902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:44.670566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46278 (local address 127.0.97.3:43181)
0926 03:09:44.671299 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:44.671313 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:44.671353 (+ 40us) server_negotiation.cc:407] Connection header received
0926 03:09:44.672959 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:44.672969 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:44.673409 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:44.673711 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:44.675156 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:44.676415 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:44.677708 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:44.678436 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:44.678701 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:44.681325 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:44.681335 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:44.681352 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:44.681407 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:44.684528 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:44.685307 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:44.685315 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:44.685324 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:44.685516 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:44.685898 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:44.685903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:44.685905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:44.686553 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:44.686848 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:44.687249 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 03:09:44.687666 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":357,"threads_started":1}
I20260926 03:09:44.689339 15844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:09:44.774869 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 15721
2026-09-26T03:09:44Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24112 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:09:44 dist-test-slave-hmbw krb5kdc[15906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:09:44 dist-test-slave-hmbw krb5kdc[15906](info): set up 2 sockets
Sep 26 03:09:44 dist-test-slave-hmbw krb5kdc[15906](info): commencing operation
krb5kdc: starting...
W20260926 03:09:45.799269 388 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:09:45 dist-test-slave-hmbw krb5kdc[15906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:09:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:09:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:09:46.042578 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39617
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:09:46.377669 15919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:09:46.377995 15919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:09:46.378132 15919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:09:46.385628 15919 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 03:09:46.385739 15919 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 03:09:46.385778 15919 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 03:09:46.385848 15919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:09:46.385921 15919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:09:46.386018 15919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:09:46.386066 15919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:09:46.386137 15919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:09:46.396579 15919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:09:46.397918 15919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:09:46.399338 15919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:09:46.411651 15927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:46.411590 15924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:46.411626 15925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:46.412286 15919 server_base.cc:1034] running on GCE node
I20260926 03:09:46.413079 15919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:09:46.414552 15919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:09:46.415817 15919 hybrid_clock.cc:648] HybridClock initialized: now 1790392186415735 us; error 52 us; skew 500 ppm
I20260926 03:09:46.416246 15919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:09:46 dist-test-slave-hmbw krb5kdc[15906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:09:46.426661 15919 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:09:46.428418 15919 webserver.cc:460] Webserver started at http://127.0.97.62:43785/ using document root <none> and password file <none>
I20260926 03:09:46.429152 15919 fs_manager.cc:359] Metadata directory not provided
I20260926 03:09:46.429261 15919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:09:46.429570 15919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:09:46.432717 15919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "cf655247a70b4c97bf4249d8e522ce54"
format_stamp: "Formatted at 2026-09-26 03:09:46 on dist-test-slave-hmbw"
server_key: "d61fd23f794e99eba896c04d9f38e2d9"
server_key_iv: "0c397aa115c0b2889dc33b34f23b0762"
server_key_version: "encryptionkey@0"
I20260926 03:09:46.433524 15919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "cf655247a70b4c97bf4249d8e522ce54"
format_stamp: "Formatted at 2026-09-26 03:09:46 on dist-test-slave-hmbw"
server_key: "d61fd23f794e99eba896c04d9f38e2d9"
server_key_iv: "0c397aa115c0b2889dc33b34f23b0762"
server_key_version: "encryptionkey@0"
I20260926 03:09:46.439143 15919 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:09:46.442873 15934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:46.444275 15919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:09:46.444442 15919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "cf655247a70b4c97bf4249d8e522ce54"
format_stamp: "Formatted at 2026-09-26 03:09:46 on dist-test-slave-hmbw"
server_key: "d61fd23f794e99eba896c04d9f38e2d9"
server_key_iv: "0c397aa115c0b2889dc33b34f23b0762"
server_key_version: "encryptionkey@0"
I20260926 03:09:46.444646 15919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:09:46.467849 15919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:09:46.472370 15919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:09:46.472637 15919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:09:46.490288 15919 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39617
I20260926 03:09:46.490306 15985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39617 every 8 connection(s)
I20260926 03:09:46.491715 15919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:09:46.495818 15986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:09:46.495790 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 15919
I20260926 03:09:46.496024 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:09:46.496896 388 external_mini_cluster.cc:1428] Setting key fc35f8155364b3c182bcea67b512c8f3
I20260926 03:09:46.511281 15986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54: Bootstrap starting.
Sep 26 03:09:46 dist-test-slave-hmbw krb5kdc[15906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:09:46.515492 15986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54: Neither blocks nor log segments found. Creating new log.
I20260926 03:09:46.516700 15986 log.cc:824] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54: Log is configured to *not* fsync() on all Append() calls
I20260926 03:09:46.520442 15986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54: No bootstrap required, opened a new log
I20260926 03:09:46.524601 15989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:46.499268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34056 (local address 127.0.97.62:39617)
0926 03:09:46.500280 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:46.500293 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:46.500336 (+ 43us) server_negotiation.cc:407] Connection header received
0926 03:09:46.502843 (+ 2507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:46.502876 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:46.503374 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:46.503762 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:46.505608 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:46.507194 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:46.509236 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:46.510152 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:46.510321 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:46.515466 (+ 5145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:46.515480 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:46.515497 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:46.515544 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:46.519388 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:46.520252 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:46.520261 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:46.520270 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:46.520505 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:46.520973 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:46.520979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:46.520982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:46.521738 (+ 756us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:46.522041 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:46.522616 (+ 575us) server_negotiation.cc:299] Negotiation successful
0926 03:09:46.523104 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":786,"thread_start_us":533,"threads_started":1}
I20260926 03:09:46.529780 15986 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf655247a70b4c97bf4249d8e522ce54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39617 } }
I20260926 03:09:46.530198 15986 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:09:46.530277 15986 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf655247a70b4c97bf4249d8e522ce54, State: Initialized, Role: FOLLOWER
I20260926 03:09:46.530920 15986 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf655247a70b4c97bf4249d8e522ce54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39617 } }
I20260926 03:09:46.531167 15986 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:09:46.531298 15986 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:09:46.531466 15986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:09:46.533912 15986 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf655247a70b4c97bf4249d8e522ce54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39617 } }
I20260926 03:09:46.534474 15986 leader_election.cc:302] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf655247a70b4c97bf4249d8e522ce54; no voters:
I20260926 03:09:46.535194 15986 leader_election.cc:288] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:09:46.535351 15991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:09:46.536414 15991 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [term 1 LEADER]: Becoming Leader. State: Replica: cf655247a70b4c97bf4249d8e522ce54, State: Running, Role: LEADER
I20260926 03:09:46.537091 15991 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf655247a70b4c97bf4249d8e522ce54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39617 } }
I20260926 03:09:46.537514 15986 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:09:46.539973 15993 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf655247a70b4c97bf4249d8e522ce54. Latest consensus state: current_term: 1 leader_uuid: "cf655247a70b4c97bf4249d8e522ce54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf655247a70b4c97bf4249d8e522ce54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39617 } } }
I20260926 03:09:46.540447 15993 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [sys.catalog]: This master's current role is: LEADER
I20260926 03:09:46.541077 15992 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf655247a70b4c97bf4249d8e522ce54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf655247a70b4c97bf4249d8e522ce54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39617 } } }
I20260926 03:09:46.541412 15992 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [sys.catalog]: This master's current role is: LEADER
I20260926 03:09:46.543309 16000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:09:46.547859 16000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:09:46.558852 16000 catalog_manager.cc:1347] Generated new cluster ID: d6a6ca50235840d78b926b99984f1578
I20260926 03:09:46.559144 16000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:09:46.567603 16000 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:09:46.569093 16000 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:09:46.577841 16000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54: Generated new TSK 0
I20260926 03:09:46.578791 16000 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:09:46.647351 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39617
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:09:46.980283 16012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:09:46.980588 16012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:09:46.980705 16012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:09:46.988359 16012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:09:46.988479 16012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:09:46.988631 16012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:09:46.998934 16012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:09:47.000276 16012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:09:47.001574 16012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:09:47.016409 16018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:47.016362 16020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:47.016439 16017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:47.016984 16012 server_base.cc:1034] running on GCE node
I20260926 03:09:47.017858 16012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:09:47.018996 16012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:09:47.020255 16012 hybrid_clock.cc:648] HybridClock initialized: now 1790392187020210 us; error 38 us; skew 500 ppm
I20260926 03:09:47.020604 16012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:09:47 dist-test-slave-hmbw krb5kdc[15906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:09:47.024106 16012 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:09:47.026206 16012 webserver.cc:460] Webserver started at http://127.0.97.1:36391/ using document root <none> and password file <none>
I20260926 03:09:47.027190 16012 fs_manager.cc:359] Metadata directory not provided
I20260926 03:09:47.027333 16012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:09:47.027669 16012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:09:47.031301 16012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/data/instance:
uuid: "67353804c3504e4287cd5aa5efe6e8ac"
format_stamp: "Formatted at 2026-09-26 03:09:47 on dist-test-slave-hmbw"
server_key: "ca21cb183e6400736abd173e7a46aa2f"
server_key_iv: "8ab072d0d9fbd1518980f9f6ed45fe0e"
server_key_version: "encryptionkey@0"
I20260926 03:09:47.032254 16012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/wal/instance:
uuid: "67353804c3504e4287cd5aa5efe6e8ac"
format_stamp: "Formatted at 2026-09-26 03:09:47 on dist-test-slave-hmbw"
server_key: "ca21cb183e6400736abd173e7a46aa2f"
server_key_iv: "8ab072d0d9fbd1518980f9f6ed45fe0e"
server_key_version: "encryptionkey@0"
I20260926 03:09:47.039151 16012 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260926 03:09:47.043630 16027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:47.045491 16012 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 03:09:47.045713 16012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "67353804c3504e4287cd5aa5efe6e8ac"
format_stamp: "Formatted at 2026-09-26 03:09:47 on dist-test-slave-hmbw"
server_key: "ca21cb183e6400736abd173e7a46aa2f"
server_key_iv: "8ab072d0d9fbd1518980f9f6ed45fe0e"
server_key_version: "encryptionkey@0"
I20260926 03:09:47.045941 16012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:09:47.063166 16012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:09:47.068070 16012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:09:47.068357 16012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:09:47.069377 16012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:09:47.070791 16012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:09:47.070871 16012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:47.070997 16012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:09:47.071051 16012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:47.102887 16012 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44023
I20260926 03:09:47.102936 16139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44023 every 8 connection(s)
I20260926 03:09:47.104192 16012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:09:47.110441 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 16012
I20260926 03:09:47.110651 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:09:47.111249 388 external_mini_cluster.cc:1428] Setting key e00be132144e2a5940973d14506c8005
Sep 26 03:09:47 dist-test-slave-hmbw krb5kdc[15906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:09:47.125020 16142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:47.106855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48219 (local address 127.0.97.62:39617)
0926 03:09:47.107330 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:47.107335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:47.108381 (+ 1046us) server_negotiation.cc:407] Connection header received
0926 03:09:47.110235 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:47.110240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:47.110328 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:47.110425 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:47.112118 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:47.112912 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:47.114522 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:47.114955 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:47.115085 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:47.119212 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:47.119218 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:47.119221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:47.119259 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:47.121750 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:47.122569 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:47.122574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:47.122576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:47.122708 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:47.123159 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:47.123163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:47.123165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:47.123463 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:47.123604 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:47.124403 (+ 799us) server_negotiation.cc:299] Negotiation successful
0926 03:09:47.124664 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":234,"threads_started":1}
I20260926 03:09:47.126047 16143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:47.107409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.1:48219)
0926 03:09:47.108170 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:47.108215 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:47.109931 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:47.110718 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:47.110727 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:47.111290 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:09:47.111930 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:47.111949 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:47.113077 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:47.113081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:47.114342 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:47.114355 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:47.115205 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:47.115212 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:47.115444 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:47.116177 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:09:47.116236 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:09:47.118939 (+ 2703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:09:47.121935 (+ 2996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:47.121946 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:47.121963 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:47.122409 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:47.122821 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:47.122826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:47.122831 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:47.123041 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:47.123596 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:09:47.123605 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:09:47.124003 (+ 398us) client_negotiation.cc:769] Sending connection context
0926 03:09:47.124257 (+ 254us) client_negotiation.cc:240] Negotiation successful
0926 03:09:47.124853 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":300,"threads_started":1}
I20260926 03:09:47.127897 16140 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39617
I20260926 03:09:47.128271 16140 heartbeater.cc:464] Registering TS with master...
I20260926 03:09:47.129209 16140 heartbeater.cc:511] Master 127.0.97.62:39617 requested a full tablet report, sending...
I20260926 03:09:47.131618 15951 ts_manager.cc:194] Registered new tserver with Master: 67353804c3504e4287cd5aa5efe6e8ac (127.0.97.1:44023)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:09:47.133140 15951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:09:47.163244 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39617
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:09:47.493347 16146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:09:47.493636 16146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:09:47.493758 16146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:09:47.501358 16146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:09:47.501484 16146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:09:47.501634 16146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:09:47.512089 16146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:09:47.513408 16146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:09:47.514747 16146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:09:47.527902 16151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:47.527932 16154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:47.528090 16152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:47.528882 16146 server_base.cc:1034] running on GCE node
I20260926 03:09:47.529561 16146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:09:47.530575 16146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:09:47.531826 16146 hybrid_clock.cc:648] HybridClock initialized: now 1790392187531780 us; error 43 us; skew 500 ppm
I20260926 03:09:47.532222 16146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:09:47 dist-test-slave-hmbw krb5kdc[15906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:09:47.535509 16146 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:09:47.537017 16146 webserver.cc:460] Webserver started at http://127.0.97.2:43237/ using document root <none> and password file <none>
I20260926 03:09:47.537777 16146 fs_manager.cc:359] Metadata directory not provided
I20260926 03:09:47.537957 16146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:09:47.538286 16146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:09:47.541723 16146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/data/instance:
uuid: "ff233c21ff6f4757b49bfe0ad0c65e33"
format_stamp: "Formatted at 2026-09-26 03:09:47 on dist-test-slave-hmbw"
server_key: "28004096e3ce5fc1033a5692df1ba6fc"
server_key_iv: "235d92060848149ef7af1c37619c51b7"
server_key_version: "encryptionkey@0"
I20260926 03:09:47.542557 16146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ff233c21ff6f4757b49bfe0ad0c65e33"
format_stamp: "Formatted at 2026-09-26 03:09:47 on dist-test-slave-hmbw"
server_key: "28004096e3ce5fc1033a5692df1ba6fc"
server_key_iv: "235d92060848149ef7af1c37619c51b7"
server_key_version: "encryptionkey@0"
I20260926 03:09:47.547919 16146 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 03:09:47.551985 16161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:47.553406 16146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:09:47.553571 16146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "ff233c21ff6f4757b49bfe0ad0c65e33"
format_stamp: "Formatted at 2026-09-26 03:09:47 on dist-test-slave-hmbw"
server_key: "28004096e3ce5fc1033a5692df1ba6fc"
server_key_iv: "235d92060848149ef7af1c37619c51b7"
server_key_version: "encryptionkey@0"
I20260926 03:09:47.553768 16146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:09:47.583009 16146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:09:47.587240 16146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:09:47.587503 16146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:09:47.588459 16146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:09:47.589766 16146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:09:47.589843 16146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:47.589965 16146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:09:47.590018 16146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:47.620157 16146 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35919
I20260926 03:09:47.620225 16273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35919 every 8 connection(s)
I20260926 03:09:47.621371 16146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:09:47.627125 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 16146
I20260926 03:09:47.627322 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:09:47.627880 388 external_mini_cluster.cc:1428] Setting key 022a6abcc9e475eb29107cb8f5318cd6
Sep 26 03:09:47 dist-test-slave-hmbw krb5kdc[15906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:09:47.641736 16142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:47.623888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51277 (local address 127.0.97.62:39617)
0926 03:09:47.624154 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:47.624160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:47.625227 (+ 1067us) server_negotiation.cc:407] Connection header received
0926 03:09:47.627221 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:47.627227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:47.627325 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:47.627439 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:47.629389 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:47.630154 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:47.631770 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:47.632203 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:47.632315 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:47.635967 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:47.635973 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:47.635977 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:47.636012 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:47.638516 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:47.639276 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:47.639280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:47.639283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:47.639416 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:47.639922 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:47.639926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:47.639928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:47.640221 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:47.640363 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:47.641145 (+ 782us) server_negotiation.cc:299] Negotiation successful
0926 03:09:47.641390 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260926 03:09:47.642900 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:47.624317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.2:51277)
0926 03:09:47.625059 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:47.625104 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:47.626811 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:47.627872 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:47.627885 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:47.628623 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:09:47.629211 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:47.629230 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:47.630315 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:47.630319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:47.631618 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:47.631632 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:47.632424 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:47.632431 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:47.632661 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:47.633311 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:09:47.633335 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:09:47.635710 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:09:47.638663 (+ 2953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:47.638673 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:47.638692 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:47.639141 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:47.639537 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:47.639542 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:47.639546 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:47.639808 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:47.640356 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:09:47.640364 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:09:47.640728 (+ 364us) client_negotiation.cc:769] Sending connection context
0926 03:09:47.641026 (+ 298us) client_negotiation.cc:240] Negotiation successful
0926 03:09:47.641595 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":300,"threads_started":1}
I20260926 03:09:47.644574 16274 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39617
I20260926 03:09:47.644883 16274 heartbeater.cc:464] Registering TS with master...
I20260926 03:09:47.645593 16274 heartbeater.cc:511] Master 127.0.97.62:39617 requested a full tablet report, sending...
I20260926 03:09:47.647215 15951 ts_manager.cc:194] Registered new tserver with Master: ff233c21ff6f4757b49bfe0ad0c65e33 (127.0.97.2:35919)
I20260926 03:09:47.648453 15951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51277
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:09:47.679083 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39617
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:09:48.017822 16279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:09:48.018146 16279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:09:48.018267 16279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:09:48.025962 16279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:09:48.026096 16279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:09:48.026242 16279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:09:48.036643 16279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:09:48.038014 16279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:09:48.039469 16279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:09:48.053260 16284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:48.053347 16287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:09:48.053347 16285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:09:48.054231 16279 server_base.cc:1034] running on GCE node
I20260926 03:09:48.054929 16279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:09:48.056550 16279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:09:48.057782 16279 hybrid_clock.cc:648] HybridClock initialized: now 1790392188057714 us; error 61 us; skew 500 ppm
I20260926 03:09:48.058177 16279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:09:48 dist-test-slave-hmbw krb5kdc[15906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:09:48.061403 16279 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:09:48.062973 16279 webserver.cc:460] Webserver started at http://127.0.97.3:32921/ using document root <none> and password file <none>
I20260926 03:09:48.063683 16279 fs_manager.cc:359] Metadata directory not provided
I20260926 03:09:48.063803 16279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:09:48.064114 16279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:09:48.067139 16279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/data/instance:
uuid: "42c7171b04bf4c4d9650ef6e041aa932"
format_stamp: "Formatted at 2026-09-26 03:09:48 on dist-test-slave-hmbw"
server_key: "9ed07e54a54a972ccb99f52d5bd1ee54"
server_key_iv: "76ee9f2deb7209212432962d1354bd38"
server_key_version: "encryptionkey@0"
I20260926 03:09:48.067914 16279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/wal/instance:
uuid: "42c7171b04bf4c4d9650ef6e041aa932"
format_stamp: "Formatted at 2026-09-26 03:09:48 on dist-test-slave-hmbw"
server_key: "9ed07e54a54a972ccb99f52d5bd1ee54"
server_key_iv: "76ee9f2deb7209212432962d1354bd38"
server_key_version: "encryptionkey@0"
I20260926 03:09:48.073391 16279 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:09:48.077165 16294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:48.078737 16279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 03:09:48.078899 16279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "42c7171b04bf4c4d9650ef6e041aa932"
format_stamp: "Formatted at 2026-09-26 03:09:48 on dist-test-slave-hmbw"
server_key: "9ed07e54a54a972ccb99f52d5bd1ee54"
server_key_iv: "76ee9f2deb7209212432962d1354bd38"
server_key_version: "encryptionkey@0"
I20260926 03:09:48.079118 16279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:09:48.095196 16279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:09:48.099821 16279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:09:48.100171 16279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:09:48.101220 16279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:09:48.102772 16279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:09:48.102861 16279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:48.102985 16279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:09:48.103048 16279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:09:48.136664 16140 heartbeater.cc:503] Master 127.0.97.62:39617 was elected leader, sending a full tablet report...
I20260926 03:09:48.137131 16279 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43637
I20260926 03:09:48.137224 16406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43637 every 8 connection(s)
I20260926 03:09:48.138672 16279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:09:48.146207 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 16279
I20260926 03:09:48.146425 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:09:48.146852 388 external_mini_cluster.cc:1428] Setting key b4fa547e8f60bd06e1b3df0771fbc47e
Sep 26 03:09:48 dist-test-slave-hmbw krb5kdc[15906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:09:48.159365 16142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:48.141350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45763 (local address 127.0.97.62:39617)
0926 03:09:48.141531 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:48.141535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:48.142763 (+ 1228us) server_negotiation.cc:407] Connection header received
0926 03:09:48.144636 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:48.144642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:48.144723 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:48.144811 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:48.146340 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.147221 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.149094 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.149543 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.149674 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:48.153407 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:48.153413 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:48.153416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:48.153452 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:48.156146 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:48.156954 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:48.156959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:48.156961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:48.157106 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:48.157565 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:48.157571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:48.157573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:48.157892 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:48.158032 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:48.158897 (+ 865us) server_negotiation.cc:299] Negotiation successful
0926 03:09:48.159098 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 03:09:48.160406 16409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:48.141840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.3:45763)
0926 03:09:48.142611 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:48.142649 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:48.144324 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:48.145087 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:48.145096 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:48.145602 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:09:48.146161 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:48.146179 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.147399 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.147403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:48.148964 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:48.148977 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.149982 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.149987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:48.150243 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:48.150908 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:09:48.150932 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:09:48.153152 (+ 2220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:09:48.156299 (+ 3147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:48.156307 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:48.156321 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:48.156820 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:48.157206 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:09:48.157209 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:09:48.157211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:09:48.157405 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:09:48.158052 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:09:48.158058 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:09:48.158440 (+ 382us) client_negotiation.cc:769] Sending connection context
0926 03:09:48.158793 (+ 353us) client_negotiation.cc:240] Negotiation successful
0926 03:09:48.159237 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":287,"threads_started":1}
I20260926 03:09:48.161999 16407 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39617
I20260926 03:09:48.162330 16407 heartbeater.cc:464] Registering TS with master...
I20260926 03:09:48.163203 16407 heartbeater.cc:511] Master 127.0.97.62:39617 requested a full tablet report, sending...
I20260926 03:09:48.164698 15950 ts_manager.cc:194] Registered new tserver with Master: 42c7171b04bf4c4d9650ef6e041aa932 (127.0.97.3:43637)
I20260926 03:09:48.165547 15950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45763
I20260926 03:09:48.175491 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:09:48.192843 16142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:48.179978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34072 (local address 127.0.97.62:39617)
0926 03:09:48.180163 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:48.180169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:48.180542 (+ 373us) server_negotiation.cc:407] Connection header received
0926 03:09:48.180720 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:48.180726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:48.180798 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:48.180884 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:48.182389 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.183179 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.184574 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.184994 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.185108 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:48.187630 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:48.187635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:48.187638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:48.187675 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:48.190148 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:48.190881 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:48.190887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:48.190891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:48.191033 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:48.191378 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:48.191383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:48.191387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:48.191731 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:48.191989 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:48.192300 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 03:09:48.192536 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 03:09:48.197288 15950 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:09:48.204391 15950 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:09:48.240290 16420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:48.217176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44023 (local address 127.0.0.1:42212)
0926 03:09:48.229723 (+ 12547us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:48.229972 (+ 249us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:48.230526 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:48.231591 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:48.231599 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:48.231631 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:09:48.232071 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:48.232087 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.234092 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.234097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:48.236749 (+ 2652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:48.236769 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.239178 (+ 2409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.239184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:48.239431 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:48.239498 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 03:09:48.239626 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 03:09:48.239829 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12298,"mutex_wait_us":1,"thread_start_us":12773,"threads_started":1}
I20260926 03:09:48.240490 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:48.217636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42212 (local address 127.0.97.1:44023)
0926 03:09:48.230558 (+ 12922us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:48.230565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:48.230593 (+ 28us) server_negotiation.cc:407] Connection header received
0926 03:09:48.230715 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:48.230725 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:48.231070 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:48.231276 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:09:48.232311 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.233925 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.236960 (+ 3035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.238646 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.238770 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:48.239021 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:48.239916 (+ 895us) server_negotiation.cc:299] Negotiation successful
0926 03:09:48.240158 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12730,"thread_start_us":12569,"threads_started":1}
I20260926 03:09:48.240523 16421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:48.217493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52992 (local address 127.0.97.2:35919)
0926 03:09:48.228957 (+ 11464us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:48.228969 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:48.230969 (+ 2000us) server_negotiation.cc:407] Connection header received
0926 03:09:48.231922 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:48.231933 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:48.232194 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:48.232364 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:09:48.233269 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.234726 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.238131 (+ 3405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.239465 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.239603 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:48.239793 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:48.240064 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 03:09:48.240244 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11242,"thread_start_us":11496,"threads_started":1}
I20260926 03:09:48.241356 16419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:48.217267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35919 (local address 127.0.0.1:52992)
0926 03:09:48.230814 (+ 13547us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:48.230838 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:48.231674 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:48.232649 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:48.232654 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:48.232679 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:09:48.233076 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:48.233089 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.234942 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.234948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:48.237924 (+ 2976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:48.237939 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.239662 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.239668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:48.239845 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:48.239873 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 03:09:48.240700 (+ 827us) client_negotiation.cc:240] Negotiation successful
0926 03:09:48.240947 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13033,"mutex_wait_us":923,"thread_start_us":12046,"threads_started":1}
I20260926 03:09:48.244807 16209 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4b2a98af444e69b4bfbb32055ca58d (DEFAULT_TABLE table=test-table [id=355773a744054518a390081f71f10ec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:09:48.244843 16075 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4b2a98af444e69b4bfbb32055ca58d (DEFAULT_TABLE table=test-table [id=355773a744054518a390081f71f10ec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:09:48.246232 16209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4b2a98af444e69b4bfbb32055ca58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:09:48.246270 16075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4b2a98af444e69b4bfbb32055ca58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:09:48.246280 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:48.231205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43637 (local address 127.0.0.1:49240)
0926 03:09:48.237570 (+ 6365us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:48.237597 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:48.237731 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:48.238850 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:48.238855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:48.238882 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:09:48.239313 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:48.239325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.241299 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.241304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:48.243647 (+ 2343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:48.243664 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.245494 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.245501 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:48.245671 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:48.245706 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 03:09:48.245800 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 03:09:48.245956 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6233,"thread_start_us":6076,"threads_started":1}
I20260926 03:09:48.246393 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:48.230727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49240 (local address 127.0.97.3:43637)
0926 03:09:48.237140 (+ 6413us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:48.237150 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:48.237713 (+ 563us) server_negotiation.cc:407] Connection header received
0926 03:09:48.238017 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:48.238028 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:48.238329 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:48.238573 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:09:48.239524 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.241101 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.243839 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.245253 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.245423 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:48.245664 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:48.245885 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 03:09:48.246057 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6203,"thread_start_us":6011,"threads_started":1}
I20260926 03:09:48.250559 16342 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4b2a98af444e69b4bfbb32055ca58d (DEFAULT_TABLE table=test-table [id=355773a744054518a390081f71f10ec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:09:48.252305 16342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4b2a98af444e69b4bfbb32055ca58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:09:48.258337 16425 tablet_bootstrap.cc:492] T 6c4b2a98af444e69b4bfbb32055ca58d P ff233c21ff6f4757b49bfe0ad0c65e33: Bootstrap starting.
I20260926 03:09:48.261039 16426 tablet_bootstrap.cc:492] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac: Bootstrap starting.
I20260926 03:09:48.262526 16425 tablet_bootstrap.cc:654] T 6c4b2a98af444e69b4bfbb32055ca58d P ff233c21ff6f4757b49bfe0ad0c65e33: Neither blocks nor log segments found. Creating new log.
I20260926 03:09:48.263787 16425 log.cc:824] T 6c4b2a98af444e69b4bfbb32055ca58d P ff233c21ff6f4757b49bfe0ad0c65e33: Log is configured to *not* fsync() on all Append() calls
I20260926 03:09:48.264751 16426 tablet_bootstrap.cc:654] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac: Neither blocks nor log segments found. Creating new log.
I20260926 03:09:48.265852 16426 log.cc:824] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac: Log is configured to *not* fsync() on all Append() calls
I20260926 03:09:48.265974 16428 tablet_bootstrap.cc:492] T 6c4b2a98af444e69b4bfbb32055ca58d P 42c7171b04bf4c4d9650ef6e041aa932: Bootstrap starting.
I20260926 03:09:48.267066 16425 tablet_bootstrap.cc:492] T 6c4b2a98af444e69b4bfbb32055ca58d P ff233c21ff6f4757b49bfe0ad0c65e33: No bootstrap required, opened a new log
I20260926 03:09:48.267421 16425 ts_tablet_manager.cc:1397] T 6c4b2a98af444e69b4bfbb32055ca58d P ff233c21ff6f4757b49bfe0ad0c65e33: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260926 03:09:48.269045 16428 tablet_bootstrap.cc:654] T 6c4b2a98af444e69b4bfbb32055ca58d P 42c7171b04bf4c4d9650ef6e041aa932: Neither blocks nor log segments found. Creating new log.
I20260926 03:09:48.270002 16428 log.cc:824] T 6c4b2a98af444e69b4bfbb32055ca58d P 42c7171b04bf4c4d9650ef6e041aa932: Log is configured to *not* fsync() on all Append() calls
I20260926 03:09:48.272840 16426 tablet_bootstrap.cc:492] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac: No bootstrap required, opened a new log
I20260926 03:09:48.273022 16428 tablet_bootstrap.cc:492] T 6c4b2a98af444e69b4bfbb32055ca58d P 42c7171b04bf4c4d9650ef6e041aa932: No bootstrap required, opened a new log
I20260926 03:09:48.273120 16426 ts_tablet_manager.cc:1397] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260926 03:09:48.273274 16428 ts_tablet_manager.cc:1397] T 6c4b2a98af444e69b4bfbb32055ca58d P 42c7171b04bf4c4d9650ef6e041aa932: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260926 03:09:48.277796 16425 raft_consensus.cc:348] T 6c4b2a98af444e69b4bfbb32055ca58d P ff233c21ff6f4757b49bfe0ad0c65e33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67353804c3504e4287cd5aa5efe6e8ac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44023 } } peers { permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 } } peers { permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260926 03:09:48.278290 16425 raft_consensus.cc:374] T 6c4b2a98af444e69b4bfbb32055ca58d P ff233c21ff6f4757b49bfe0ad0c65e33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:09:48.278456 16425 raft_consensus.cc:729] T 6c4b2a98af444e69b4bfbb32055ca58d P ff233c21ff6f4757b49bfe0ad0c65e33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff233c21ff6f4757b49bfe0ad0c65e33, State: Initialized, Role: FOLLOWER
I20260926 03:09:48.279217 16425 consensus_queue.cc:260] T 6c4b2a98af444e69b4bfbb32055ca58d P ff233c21ff6f4757b49bfe0ad0c65e33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67353804c3504e4287cd5aa5efe6e8ac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44023 } } peers { permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 } } peers { permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260926 03:09:48.280506 16274 heartbeater.cc:503] Master 127.0.97.62:39617 was elected leader, sending a full tablet report...
I20260926 03:09:48.281831 16426 raft_consensus.cc:348] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67353804c3504e4287cd5aa5efe6e8ac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44023 } } peers { permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 } } peers { permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260926 03:09:48.282387 16426 raft_consensus.cc:374] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:09:48.282563 16426 raft_consensus.cc:729] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67353804c3504e4287cd5aa5efe6e8ac, State: Initialized, Role: FOLLOWER
I20260926 03:09:48.282842 16425 ts_tablet_manager.cc:1428] T 6c4b2a98af444e69b4bfbb32055ca58d P ff233c21ff6f4757b49bfe0ad0c65e33: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260926 03:09:48.283383 16426 consensus_queue.cc:260] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67353804c3504e4287cd5aa5efe6e8ac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44023 } } peers { permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 } } peers { permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260926 03:09:48.283460 16428 raft_consensus.cc:348] T 6c4b2a98af444e69b4bfbb32055ca58d P 42c7171b04bf4c4d9650ef6e041aa932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67353804c3504e4287cd5aa5efe6e8ac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44023 } } peers { permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 } } peers { permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260926 03:09:48.284062 16428 raft_consensus.cc:374] T 6c4b2a98af444e69b4bfbb32055ca58d P 42c7171b04bf4c4d9650ef6e041aa932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:09:48.284212 16428 raft_consensus.cc:729] T 6c4b2a98af444e69b4bfbb32055ca58d P 42c7171b04bf4c4d9650ef6e041aa932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42c7171b04bf4c4d9650ef6e041aa932, State: Initialized, Role: FOLLOWER
I20260926 03:09:48.284961 16428 consensus_queue.cc:260] T 6c4b2a98af444e69b4bfbb32055ca58d P 42c7171b04bf4c4d9650ef6e041aa932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67353804c3504e4287cd5aa5efe6e8ac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44023 } } peers { permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 } } peers { permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260926 03:09:48.285367 16426 ts_tablet_manager.cc:1428] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac: Time spent starting tablet: real 0.012s user 0.011s sys 0.002s
I20260926 03:09:48.288833 16407 heartbeater.cc:503] Master 127.0.97.62:39617 was elected leader, sending a full tablet report...
I20260926 03:09:48.289341 16428 ts_tablet_manager.cc:1428] T 6c4b2a98af444e69b4bfbb32055ca58d P 42c7171b04bf4c4d9650ef6e041aa932: Time spent starting tablet: real 0.016s user 0.011s sys 0.006s
I20260926 03:09:48.352312 16432 raft_consensus.cc:482] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:09:48.352612 16432 raft_consensus.cc:504] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67353804c3504e4287cd5aa5efe6e8ac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44023 } } peers { permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 } } peers { permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260926 03:09:48.354308 16432 leader_election.cc:288] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff233c21ff6f4757b49bfe0ad0c65e33 (127.0.97.2:35919), 42c7171b04bf4c4d9650ef6e041aa932 (127.0.97.3:43637)
W20260926 03:09:48.356505 16141 tablet.cc:2369] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:09:48.362116 16421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:48.354832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49875 (local address 127.0.97.2:35919)
0926 03:09:48.355042 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:48.355047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:48.355535 (+ 488us) server_negotiation.cc:407] Connection header received
0926 03:09:48.355761 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:48.355767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:48.355847 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:48.355968 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:09:48.356819 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.357694 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.359894 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.361181 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.361314 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:48.361403 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:48.361805 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 03:09:48.361924 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:09:48.362470 16434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:48.354752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35919 (local address 127.0.97.1:49875)
0926 03:09:48.355438 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:48.355457 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:48.355611 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:48.356162 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:48.356167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:48.356203 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:09:48.356666 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:48.356680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.357807 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.357812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:48.359715 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:48.359727 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.361391 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.361398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:48.361580 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:48.361615 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 03:09:48.361715 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 03:09:48.361863 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":21,"thread_start_us":394,"threads_started":1}
I20260926 03:09:48.363188 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:48.355849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43713 (local address 127.0.97.3:43637)
0926 03:09:48.356085 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:48.356091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:48.356603 (+ 512us) server_negotiation.cc:407] Connection header received
0926 03:09:48.357088 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:48.357092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:48.357151 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:48.357223 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:09:48.357732 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.358849 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.360978 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.362086 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.362206 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:48.362301 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:48.362799 (+ 498us) server_negotiation.cc:299] Negotiation successful
0926 03:09:48.362895 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:09:48.363217 16435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:48.355639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43637 (local address 127.0.97.1:43713)
0926 03:09:48.356475 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0926 03:09:48.356498 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:09:48.356899 (+ 401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:09:48.357259 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:09:48.357263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:09:48.357286 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:09:48.357593 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:48.357602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.359016 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.359020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:48.360796 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:09:48.360811 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.362311 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.362318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:09:48.362490 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:48.362546 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 03:09:48.362775 (+ 229us) client_negotiation.cc:240] Negotiation successful
0926 03:09:48.362903 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":2,"thread_start_us":450,"threads_started":1}
I20260926 03:09:48.363600 16229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4b2a98af444e69b4bfbb32055ca58d" candidate_uuid: "67353804c3504e4287cd5aa5efe6e8ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" is_pre_election: true
I20260926 03:09:48.364145 16229 raft_consensus.cc:2454] T 6c4b2a98af444e69b4bfbb32055ca58d P ff233c21ff6f4757b49bfe0ad0c65e33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67353804c3504e4287cd5aa5efe6e8ac in term 0.
I20260926 03:09:48.364123 16362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4b2a98af444e69b4bfbb32055ca58d" candidate_uuid: "67353804c3504e4287cd5aa5efe6e8ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42c7171b04bf4c4d9650ef6e041aa932" is_pre_election: true
I20260926 03:09:48.364625 16362 raft_consensus.cc:2454] T 6c4b2a98af444e69b4bfbb32055ca58d P 42c7171b04bf4c4d9650ef6e041aa932 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67353804c3504e4287cd5aa5efe6e8ac in term 0.
I20260926 03:09:48.364777 16030 leader_election.cc:302] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67353804c3504e4287cd5aa5efe6e8ac, ff233c21ff6f4757b49bfe0ad0c65e33; no voters:
I20260926 03:09:48.365182 16432 raft_consensus.cc:2781] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:09:48.365314 16432 raft_consensus.cc:482] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:09:48.365423 16432 raft_consensus.cc:3037] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:09:48.367694 16432 raft_consensus.cc:504] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67353804c3504e4287cd5aa5efe6e8ac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44023 } } peers { permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 } } peers { permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260926 03:09:48.368280 16432 leader_election.cc:288] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 1 election: Requested vote from peers ff233c21ff6f4757b49bfe0ad0c65e33 (127.0.97.2:35919), 42c7171b04bf4c4d9650ef6e041aa932 (127.0.97.3:43637)
I20260926 03:09:48.368857 16229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4b2a98af444e69b4bfbb32055ca58d" candidate_uuid: "67353804c3504e4287cd5aa5efe6e8ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33"
I20260926 03:09:48.368963 16362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4b2a98af444e69b4bfbb32055ca58d" candidate_uuid: "67353804c3504e4287cd5aa5efe6e8ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42c7171b04bf4c4d9650ef6e041aa932"
I20260926 03:09:48.369113 16229 raft_consensus.cc:3037] T 6c4b2a98af444e69b4bfbb32055ca58d P ff233c21ff6f4757b49bfe0ad0c65e33 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:09:48.369156 16362 raft_consensus.cc:3037] T 6c4b2a98af444e69b4bfbb32055ca58d P 42c7171b04bf4c4d9650ef6e041aa932 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:09:48.371364 16229 raft_consensus.cc:2454] T 6c4b2a98af444e69b4bfbb32055ca58d P ff233c21ff6f4757b49bfe0ad0c65e33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67353804c3504e4287cd5aa5efe6e8ac in term 1.
I20260926 03:09:48.371371 16362 raft_consensus.cc:2454] T 6c4b2a98af444e69b4bfbb32055ca58d P 42c7171b04bf4c4d9650ef6e041aa932 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67353804c3504e4287cd5aa5efe6e8ac in term 1.
I20260926 03:09:48.371789 16030 leader_election.cc:302] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67353804c3504e4287cd5aa5efe6e8ac, ff233c21ff6f4757b49bfe0ad0c65e33; no voters:
I20260926 03:09:48.372102 16432 raft_consensus.cc:2781] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:09:48.372534 16432 raft_consensus.cc:686] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac [term 1 LEADER]: Becoming Leader. State: Replica: 67353804c3504e4287cd5aa5efe6e8ac, State: Running, Role: LEADER
I20260926 03:09:48.373059 16432 consensus_queue.cc:237] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67353804c3504e4287cd5aa5efe6e8ac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44023 } } peers { permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 } } peers { permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
W20260926 03:09:48.373260 16275 tablet.cc:2369] T 6c4b2a98af444e69b4bfbb32055ca58d P ff233c21ff6f4757b49bfe0ad0c65e33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:09:48.377568 15950 catalog_manager.cc:5495] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 67353804c3504e4287cd5aa5efe6e8ac (127.0.97.1). New cstate: current_term: 1 leader_uuid: "67353804c3504e4287cd5aa5efe6e8ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67353804c3504e4287cd5aa5efe6e8ac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } health_report { overall_health: UNKNOWN } } }
W20260926 03:09:48.390336 16408 tablet.cc:2369] T 6c4b2a98af444e69b4bfbb32055ca58d P 42c7171b04bf4c4d9650ef6e041aa932: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:09:48.403841 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:48.398037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42220 (local address 127.0.97.1:44023)
0926 03:09:48.398210 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:48.398214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:48.398325 (+ 111us) server_negotiation.cc:407] Connection header received
0926 03:09:48.398562 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:48.398566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:48.398667 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:48.398790 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:09:48.399133 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.399829 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.401549 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.401979 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.402081 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:48.402606 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:09:48.403251 (+ 645us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:09:48.403371 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:48.403550 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 03:09:48.403637 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 03:09:48.412403 16229 raft_consensus.cc:1261] T 6c4b2a98af444e69b4bfbb32055ca58d P ff233c21ff6f4757b49bfe0ad0c65e33 [term 1 FOLLOWER]: Refusing update from remote peer 67353804c3504e4287cd5aa5efe6e8ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:09:48.412415 16362 raft_consensus.cc:1261] T 6c4b2a98af444e69b4bfbb32055ca58d P 42c7171b04bf4c4d9650ef6e041aa932 [term 1 FOLLOWER]: Refusing update from remote peer 67353804c3504e4287cd5aa5efe6e8ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:09:48.413434 16436 consensus_queue.cc:1035] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:09:48.413779 16432 consensus_queue.cc:1035] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:09:48.425841 16441 mvcc.cc:205] Tried to move back new op lower bound from 7333446403725930496 to 7333446403579809792. Current Snapshot: MvccSnapshot[applied={T|T < 7333446403725930496}]
I20260926 03:09:48.432120 16439 mvcc.cc:205] Tried to move back new op lower bound from 7333446403725930496 to 7333446403579809792. Current Snapshot: MvccSnapshot[applied={T|T < 7333446403725930496 or (T in {7333446403725930496})}]
I20260926 03:09:48.438288 16440 mvcc.cc:205] Tried to move back new op lower bound from 7333446403725930496 to 7333446403579809792. Current Snapshot: MvccSnapshot[applied={T|T < 7333446403725930496}]
I20260926 03:09:48.442582 15949 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:09:48.442875 15949 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:09:48.446404 15949 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54: Sending DeleteTablet for 3 replicas of tablet 6c4b2a98af444e69b4bfbb32055ca58d
I20260926 03:09:48.447733 16075 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4b2a98af444e69b4bfbb32055ca58d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:09:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42212
I20260926 03:09:48.447785 16342 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4b2a98af444e69b4bfbb32055ca58d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:09:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49240
I20260926 03:09:48.447785 16209 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4b2a98af444e69b4bfbb32055ca58d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:09:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52992
I20260926 03:09:48.449241 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 16012
I20260926 03:09:48.476001 16452 tablet_replica.cc:330] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac: stopping tablet replica
I20260926 03:09:48.476653 16452 raft_consensus.cc:2229] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:09:48.479176 16452 raft_consensus.cc:2258] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:09:48.482734 16451 tablet_replica.cc:330] T 6c4b2a98af444e69b4bfbb32055ca58d P ff233c21ff6f4757b49bfe0ad0c65e33: stopping tablet replica
I20260926 03:09:48.483259 16451 raft_consensus.cc:2229] T 6c4b2a98af444e69b4bfbb32055ca58d P ff233c21ff6f4757b49bfe0ad0c65e33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:09:48.483937 16451 raft_consensus.cc:2258] T 6c4b2a98af444e69b4bfbb32055ca58d P ff233c21ff6f4757b49bfe0ad0c65e33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:09:48.484155 16450 tablet_replica.cc:330] T 6c4b2a98af444e69b4bfbb32055ca58d P 42c7171b04bf4c4d9650ef6e041aa932: stopping tablet replica
I20260926 03:09:48.485201 16452 ts_tablet_manager.cc:1905] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:09:48.485787 16451 ts_tablet_manager.cc:1905] T 6c4b2a98af444e69b4bfbb32055ca58d P ff233c21ff6f4757b49bfe0ad0c65e33: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 03:09:48 dist-test-slave-hmbw krb5kdc[15906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:09:48.488525 16450 raft_consensus.cc:2229] T 6c4b2a98af444e69b4bfbb32055ca58d P 42c7171b04bf4c4d9650ef6e041aa932 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:09:48.490559 16450 raft_consensus.cc:2258] T 6c4b2a98af444e69b4bfbb32055ca58d P 42c7171b04bf4c4d9650ef6e041aa932 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:09:48.492725 16451 ts_tablet_manager.cc:1918] T 6c4b2a98af444e69b4bfbb32055ca58d P ff233c21ff6f4757b49bfe0ad0c65e33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:09:48.492725 16452 ts_tablet_manager.cc:1918] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:09:48.492905 16451 log.cc:1196] T 6c4b2a98af444e69b4bfbb32055ca58d P ff233c21ff6f4757b49bfe0ad0c65e33: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/wal/wals/6c4b2a98af444e69b4bfbb32055ca58d
I20260926 03:09:48.492902 16452 log.cc:1196] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/wal/wals/6c4b2a98af444e69b4bfbb32055ca58d
I20260926 03:09:48.493431 16452 ts_tablet_manager.cc:1939] T 6c4b2a98af444e69b4bfbb32055ca58d P 67353804c3504e4287cd5aa5efe6e8ac: Deleting consensus metadata
I20260926 03:09:48.493443 16451 ts_tablet_manager.cc:1939] T 6c4b2a98af444e69b4bfbb32055ca58d P ff233c21ff6f4757b49bfe0ad0c65e33: Deleting consensus metadata
I20260926 03:09:48.492784 16450 ts_tablet_manager.cc:1905] T 6c4b2a98af444e69b4bfbb32055ca58d P 42c7171b04bf4c4d9650ef6e041aa932: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:09:48.495487 15937 catalog_manager.cc:4841] TS ff233c21ff6f4757b49bfe0ad0c65e33 (127.0.97.2:35919): tablet 6c4b2a98af444e69b4bfbb32055ca58d (table test-table [id=355773a744054518a390081f71f10ec6]) successfully deleted
I20260926 03:09:48.495873 15936 catalog_manager.cc:4841] TS 67353804c3504e4287cd5aa5efe6e8ac (127.0.97.1:44023): tablet 6c4b2a98af444e69b4bfbb32055ca58d (table test-table [id=355773a744054518a390081f71f10ec6]) successfully deleted
I20260926 03:09:48.501255 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:48.450655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42224 (local address 127.0.97.1:44023)
0926 03:09:48.450860 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:48.450867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:48.472648 (+ 21781us) server_negotiation.cc:407] Connection header received
0926 03:09:48.473206 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:48.473212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:48.473314 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:48.473408 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:48.477067 (+ 3659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.478365 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.482071 (+ 3706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.482808 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.482968 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:48.493128 (+ 10160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:48.493146 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:48.493164 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:48.493219 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:48.497307 (+ 4088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:48.498381 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:48.498391 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:48.498395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:48.498608 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:48.499080 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:48.499085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:48.499089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:48.499554 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:48.499825 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:48.500489 (+ 664us) server_negotiation.cc:299] Negotiation successful
0926 03:09:48.500744 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:09:48.503175 16135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:09:48.505213 16450 ts_tablet_manager.cc:1918] T 6c4b2a98af444e69b4bfbb32055ca58d P 42c7171b04bf4c4d9650ef6e041aa932: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:09:48.505424 16450 log.cc:1196] T 6c4b2a98af444e69b4bfbb32055ca58d P 42c7171b04bf4c4d9650ef6e041aa932: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/wal/wals/6c4b2a98af444e69b4bfbb32055ca58d
I20260926 03:09:48.505988 16450 ts_tablet_manager.cc:1939] T 6c4b2a98af444e69b4bfbb32055ca58d P 42c7171b04bf4c4d9650ef6e041aa932: Deleting consensus metadata
I20260926 03:09:48.508183 15937 catalog_manager.cc:4841] TS 42c7171b04bf4c4d9650ef6e041aa932 (127.0.97.3:43637): tablet 6c4b2a98af444e69b4bfbb32055ca58d (table test-table [id=355773a744054518a390081f71f10ec6]) successfully deleted
I20260926 03:09:48.606104 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 16012
I20260926 03:09:48.621641 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 16146
Sep 26 03:09:48 dist-test-slave-hmbw krb5kdc[15906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:09:48.637825 16421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:48.622467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53006 (local address 127.0.97.2:35919)
0926 03:09:48.622653 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:48.622658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:48.622677 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:09:48.622870 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:48.622877 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:48.622955 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:48.623060 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:48.624423 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.625191 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.626572 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.627042 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.627163 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:48.631656 (+ 4493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:48.631669 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:48.631684 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:48.631726 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:48.634915 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:48.635721 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:48.635741 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:48.635744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:48.635889 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:48.636246 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:48.636251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:48.636253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:48.636656 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:48.636940 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:48.637280 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 03:09:48.637468 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:09:48.638598 16269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:09:48.728185 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 16146
I20260926 03:09:48.742547 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 16279
Sep 26 03:09:48 dist-test-slave-hmbw krb5kdc[15906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:09:48.759030 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:09:48.743256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49250 (local address 127.0.97.3:43637)
0926 03:09:48.743405 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 03:09:48.743409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:09:48.743450 (+ 41us) server_negotiation.cc:407] Connection header received
0926 03:09:48.743560 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:09:48.743564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:09:48.743630 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:09:48.743713 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:09:48.745045 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.745813 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.747691 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:09:48.748324 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:09:48.748473 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:09:48.753383 (+ 4910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:09:48.753405 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:09:48.753423 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:09:48.753487 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:09:48.756416 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:48.757172 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:48.757179 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:48.757181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:48.757331 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:09:48.757656 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:09:48.757659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:09:48.757662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:09:48.758033 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:09:48.758248 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 03:09:48.758597 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 03:09:48.758777 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 03:09:48.759680 16402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:09:48.847762 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 16279
I20260926 03:09:48.861366 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 15919
I20260926 03:09:48.862679 15981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:09:48.974161 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 15919
I20260926 03:10:08.992549 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39617
--webserver_interface=127.0.97.62
--webserver_port=43785
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:10:09.376729 16458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:10:09.377084 16458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:10:09.377233 16458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:10:09.384793 16458 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 03:10:09.384912 16458 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 03:10:09.384950 16458 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 03:10:09.385016 16458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:10:09.385085 16458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:10:09.385175 16458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:10:09.385231 16458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:10:09.385305 16458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:10:09.395668 16458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=43785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:10:09.396991 16458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:10:09.398304 16458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:10:09.421998 16464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:09.421947 16465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:09.421976 16467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:10:09.422226 16458 server_base.cc:1034] running on GCE node
I20260926 03:10:09.423532 16458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:10:09.425218 16458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:10:09.426484 16458 hybrid_clock.cc:648] HybridClock initialized: now 1790392209426409 us; error 62 us; skew 500 ppm
I20260926 03:10:09.427016 16458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:10:09 dist-test-slave-hmbw krb5kdc[15906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:10:09.438757 16458 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:10:09.440747 16458 webserver.cc:460] Webserver started at http://127.0.97.62:43785/ using document root <none> and password file <none>
I20260926 03:10:09.441612 16458 fs_manager.cc:359] Metadata directory not provided
I20260926 03:10:09.441746 16458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:10:09.448232 16458 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 03:10:09.451782 16474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:09.453353 16458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:10:09.453549 16458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "cf655247a70b4c97bf4249d8e522ce54"
format_stamp: "Formatted at 2026-09-26 03:09:46 on dist-test-slave-hmbw"
server_key: "d61fd23f794e99eba896c04d9f38e2d9"
server_key_iv: "0c397aa115c0b2889dc33b34f23b0762"
server_key_version: "encryptionkey@0"
I20260926 03:10:09.454728 16458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:10:09.478128 16458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:10:09.482916 16458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:10:09.483222 16458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:10:09.501724 16458 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39617
I20260926 03:10:09.501744 16525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39617 every 8 connection(s)
I20260926 03:10:09.503413 16458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:10:09.510844 16526 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:10:09.512852 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 16458
I20260926 03:10:09.513623 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:44023
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:39617
--webserver_port=36391
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:10:09.514429 16526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54: Bootstrap starting.
I20260926 03:10:09.534077 16526 log.cc:824] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54: Log is configured to *not* fsync() on all Append() calls
I20260926 03:10:09.546959 16526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:10:09.547519 16526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54: Bootstrap complete.
I20260926 03:10:09.556891 16526 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf655247a70b4c97bf4249d8e522ce54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39617 } }
I20260926 03:10:09.557905 16526 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf655247a70b4c97bf4249d8e522ce54, State: Initialized, Role: FOLLOWER
I20260926 03:10:09.558643 16526 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf655247a70b4c97bf4249d8e522ce54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39617 } }
I20260926 03:10:09.558916 16526 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:10:09.559041 16526 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:10:09.559201 16526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:10:09.563408 16526 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf655247a70b4c97bf4249d8e522ce54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39617 } }
I20260926 03:10:09.563984 16526 leader_election.cc:302] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf655247a70b4c97bf4249d8e522ce54; no voters:
I20260926 03:10:09.564635 16526 leader_election.cc:288] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 03:10:09.564769 16530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 03:10:09.565889 16530 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [term 2 LEADER]: Becoming Leader. State: Replica: cf655247a70b4c97bf4249d8e522ce54, State: Running, Role: LEADER
I20260926 03:10:09.566566 16530 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf655247a70b4c97bf4249d8e522ce54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39617 } }
I20260926 03:10:09.567039 16526 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:10:09.569032 16532 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf655247a70b4c97bf4249d8e522ce54. Latest consensus state: current_term: 2 leader_uuid: "cf655247a70b4c97bf4249d8e522ce54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf655247a70b4c97bf4249d8e522ce54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39617 } } }
I20260926 03:10:09.569454 16532 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [sys.catalog]: This master's current role is: LEADER
I20260926 03:10:09.570145 16531 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cf655247a70b4c97bf4249d8e522ce54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf655247a70b4c97bf4249d8e522ce54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39617 } } }
I20260926 03:10:09.570487 16531 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [sys.catalog]: This master's current role is: LEADER
I20260926 03:10:09.575554 16537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:10:09.584903 16537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:10:09.586867 16537 catalog_manager.cc:1259] Loaded cluster ID: d6a6ca50235840d78b926b99984f1578
I20260926 03:10:09.587072 16537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:10:09.591857 16537 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:10:09.593953 16537 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54: Loaded TSK: 0
I20260926 03:10:09.612418 16547 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 03:10:09.630031 16537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54: Generated new TSK 1
I20260926 03:10:09.630548 16537 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 03:10:09.902876 16528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:10:09.903198 16528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:10:09.903342 16528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:10:09.910955 16528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:10:09.911099 16528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:10:09.911267 16528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:10:09.922322 16528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:44023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36391
--tserver_master_addrs=127.0.97.62:39617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:10:09.923784 16528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:10:09.925433 16528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:10:09.939965 16553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:09.940078 16556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:09.940480 16554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:10:09.941089 16528 server_base.cc:1034] running on GCE node
I20260926 03:10:09.941885 16528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:10:09.943192 16528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:10:09.944448 16528 hybrid_clock.cc:648] HybridClock initialized: now 1790392209944398 us; error 40 us; skew 500 ppm
I20260926 03:10:09.944861 16528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:10:09 dist-test-slave-hmbw krb5kdc[15906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:10:09.949118 16528 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:10:09.951216 16528 webserver.cc:460] Webserver started at http://127.0.97.1:36391/ using document root <none> and password file <none>
I20260926 03:10:09.952189 16528 fs_manager.cc:359] Metadata directory not provided
I20260926 03:10:09.952350 16528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:10:09.960312 16528 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:10:09.964011 16563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:09.965852 16528 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 03:10:09.966050 16528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "67353804c3504e4287cd5aa5efe6e8ac"
format_stamp: "Formatted at 2026-09-26 03:09:47 on dist-test-slave-hmbw"
server_key: "ca21cb183e6400736abd173e7a46aa2f"
server_key_iv: "8ab072d0d9fbd1518980f9f6ed45fe0e"
server_key_version: "encryptionkey@0"
I20260926 03:10:09.967482 16528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:10:10.007230 16528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:10:10.013296 16528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:10:10.013612 16528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:10:10.014801 16528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:10:10.017629 16528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:10:10.017768 16528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:10.017897 16528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:10:10.017969 16528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:10.052547 16528 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44023
I20260926 03:10:10.052616 16675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44023 every 8 connection(s)
I20260926 03:10:10.054136 16528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:10:10.059934 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 16528
I20260926 03:10:10.060537 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:35919
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:39617
--webserver_port=43237
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:10:10 dist-test-slave-hmbw krb5kdc[15906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:10:10.082018 16679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:10.057771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.1:40913)
0926 03:10:10.058641 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:10.058682 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:10.060544 (+ 1862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:10.062290 (+ 1746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:10.062299 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:10.062917 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:10:10.063489 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:10.063508 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:10.066050 (+ 2542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:10.066055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:10.067562 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:10.067581 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:10.068893 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:10.068902 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:10.069209 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:10.069928 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:10:10.069953 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:10:10.072269 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:10:10.076741 (+ 4472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:10:10.076749 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:10:10.076767 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:10:10.077433 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:10:10.078066 (+ 633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:10:10.078071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:10:10.078074 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:10:10.078335 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:10:10.079643 (+ 1308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:10:10.079655 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:10:10.080086 (+ 431us) client_negotiation.cc:769] Sending connection context
0926 03:10:10.080355 (+ 269us) client_negotiation.cc:240] Negotiation successful
0926 03:10:10.080815 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":358,"threads_started":1}
I20260926 03:10:10.082401 16678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:10.057410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40913 (local address 127.0.97.62:39617)
0926 03:10:10.058203 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:10.058222 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:10.058820 (+ 598us) server_negotiation.cc:407] Connection header received
0926 03:10:10.061157 (+ 2337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:10.061176 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:10.061674 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:10.062095 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:10.063806 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:10.065774 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:10.067761 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:10.068586 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:10.068749 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:10.072659 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:10.072671 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:10.072693 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:10:10.072751 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:10.076537 (+ 3786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:10.077604 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:10.077614 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:10.077621 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:10.077850 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:10.078637 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:10.078643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:10.078647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:10.079454 (+ 807us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:10.079667 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:10.080664 (+ 997us) server_negotiation.cc:299] Negotiation successful
0926 03:10:10.081170 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":318,"threads_started":1}
I20260926 03:10:10.086066 16676 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39617
I20260926 03:10:10.086726 16676 heartbeater.cc:464] Registering TS with master...
I20260926 03:10:10.087924 16676 heartbeater.cc:511] Master 127.0.97.62:39617 requested a full tablet report, sending...
I20260926 03:10:10.091063 16491 ts_manager.cc:194] Registered new tserver with Master: 67353804c3504e4287cd5aa5efe6e8ac (127.0.97.1:44023)
I20260926 03:10:10.093190 16491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40913
W20260926 03:10:10.419418 16680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:10:10.419832 16680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:10:10.419970 16680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:10:10.427345 16680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:10:10.427479 16680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:10:10.427639 16680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:10:10.438007 16680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:35919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43237
--tserver_master_addrs=127.0.97.62:39617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:10:10.439416 16680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:10:10.440857 16680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:10:10.455883 16686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:10.455833 16685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:10.455858 16688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:10:10.456712 16680 server_base.cc:1034] running on GCE node
I20260926 03:10:10.457342 16680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:10:10.458446 16680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:10:10.459756 16680 hybrid_clock.cc:648] HybridClock initialized: now 1790392210459656 us; error 51 us; skew 500 ppm
I20260926 03:10:10.460071 16680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:10:10 dist-test-slave-hmbw krb5kdc[15906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:10:10.463184 16680 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:10:10.464699 16680 webserver.cc:460] Webserver started at http://127.0.97.2:43237/ using document root <none> and password file <none>
I20260926 03:10:10.465395 16680 fs_manager.cc:359] Metadata directory not provided
I20260926 03:10:10.465531 16680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:10:10.471261 16680 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 03:10:10.474190 16695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:10.475785 16680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:10:10.475960 16680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "ff233c21ff6f4757b49bfe0ad0c65e33"
format_stamp: "Formatted at 2026-09-26 03:09:47 on dist-test-slave-hmbw"
server_key: "28004096e3ce5fc1033a5692df1ba6fc"
server_key_iv: "235d92060848149ef7af1c37619c51b7"
server_key_version: "encryptionkey@0"
I20260926 03:10:10.476986 16680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:10:10.492479 16680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:10:10.496934 16680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:10:10.497206 16680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:10:10.498241 16680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:10:10.499612 16680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:10:10.499717 16680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:10.499833 16680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:10:10.499902 16680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:10.532007 16680 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35919
I20260926 03:10:10.532063 16807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35919 every 8 connection(s)
I20260926 03:10:10.533596 16680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:10:10.539171 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 16680
I20260926 03:10:10.539896 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:43637
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:39617
--webserver_port=32921
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:10:10 dist-test-slave-hmbw krb5kdc[15906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:10:10.557461 16678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:10.536551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52535 (local address 127.0.97.62:39617)
0926 03:10:10.536832 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:10.536837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:10.537953 (+ 1116us) server_negotiation.cc:407] Connection header received
0926 03:10:10.540451 (+ 2498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:10.540456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:10.540557 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:10.540648 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:10.542756 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:10.543500 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:10.545787 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:10.546253 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:10.546363 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:10.550425 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:10.550432 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:10.550436 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:10:10.550485 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:10.553855 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:10.554943 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:10.554948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:10.554952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:10.555121 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:10.555614 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:10.555618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:10.555621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:10.556000 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:10.556144 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:10.556873 (+ 729us) server_negotiation.cc:299] Negotiation successful
0926 03:10:10.557138 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260926 03:10:10.558328 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:10.537020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.2:52535)
0926 03:10:10.537777 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:10.537821 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:10.540106 (+ 2285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:10.540933 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:10.540946 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:10.541673 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:10:10.542457 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:10.542483 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:10.543679 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:10.543683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:10.545556 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:10.545571 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:10.546533 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:10.546541 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:10.546839 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:10.547612 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:10:10.547637 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:10:10.550101 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:10:10.554068 (+ 3967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:10:10.554077 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:10:10.554095 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:10:10.554782 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:10:10.555276 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:10:10.555280 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:10:10.555282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:10:10.555494 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:10:10.556119 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:10:10.556126 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:10:10.556504 (+ 378us) client_negotiation.cc:769] Sending connection context
0926 03:10:10.556758 (+ 254us) client_negotiation.cc:240] Negotiation successful
0926 03:10:10.557190 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":292,"threads_started":1}
I20260926 03:10:10.559928 16808 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39617
I20260926 03:10:10.560237 16808 heartbeater.cc:464] Registering TS with master...
I20260926 03:10:10.560966 16808 heartbeater.cc:511] Master 127.0.97.62:39617 requested a full tablet report, sending...
I20260926 03:10:10.562440 16491 ts_manager.cc:194] Registered new tserver with Master: ff233c21ff6f4757b49bfe0ad0c65e33 (127.0.97.2:35919)
I20260926 03:10:10.563279 16491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52535
W20260926 03:10:10.873570 16811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:10:10.873859 16811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:10:10.873980 16811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:10:10.881477 16811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:10:10.881598 16811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:10:10.881759 16811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:10:10.891973 16811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:43637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=32921
--tserver_master_addrs=127.0.97.62:39617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:10:10.893293 16811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:10:10.894678 16811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:10:10.908315 16817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:10.908268 16819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:10.908344 16816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:10:10.914924 16811 server_base.cc:1034] running on GCE node
I20260926 03:10:10.915652 16811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:10:10.916782 16811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:10:10.918008 16811 hybrid_clock.cc:648] HybridClock initialized: now 1790392210917961 us; error 42 us; skew 500 ppm
I20260926 03:10:10.918335 16811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:10:10 dist-test-slave-hmbw krb5kdc[15906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:10:10.921449 16811 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:10:10.923045 16811 webserver.cc:460] Webserver started at http://127.0.97.3:32921/ using document root <none> and password file <none>
I20260926 03:10:10.923750 16811 fs_manager.cc:359] Metadata directory not provided
I20260926 03:10:10.923867 16811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:10:10.929723 16811 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 03:10:10.932525 16826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:10.933981 16811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:10:10.934136 16811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "42c7171b04bf4c4d9650ef6e041aa932"
format_stamp: "Formatted at 2026-09-26 03:09:48 on dist-test-slave-hmbw"
server_key: "9ed07e54a54a972ccb99f52d5bd1ee54"
server_key_iv: "76ee9f2deb7209212432962d1354bd38"
server_key_version: "encryptionkey@0"
I20260926 03:10:10.935173 16811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:10:10.960209 16811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:10:10.964677 16811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:10:10.964960 16811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:10:10.966034 16811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:10:10.967520 16811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:10:10.967617 16811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:10.967724 16811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:10:10.967792 16811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:10.999472 16811 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43637
I20260926 03:10:10.999529 16938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43637 every 8 connection(s)
I20260926 03:10:11.000900 16811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:10:11.008121 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 16811
Sep 26 03:10:11 dist-test-slave-hmbw krb5kdc[15906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:10:11.023598 16678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:11.003501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52837 (local address 127.0.97.62:39617)
0926 03:10:11.003740 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:11.003747 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:11.004956 (+ 1209us) server_negotiation.cc:407] Connection header received
0926 03:10:11.006900 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:11.006905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:11.006980 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:11.007070 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:11.008948 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.010066 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.011899 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.012387 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.012492 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:11.016737 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:11.016744 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:11.016749 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:10:11.016793 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:11.020065 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:11.020912 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:11.020916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:11.020919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:11.021044 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:11.021520 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:11.021523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:11.021526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:11.021820 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:11.021933 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:11.022943 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0926 03:10:11.023193 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"spinlock_wait_cycles":2432}
I20260926 03:10:11.024369 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:11.009234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43996 (local address 127.0.97.62:39617)
0926 03:10:11.009951 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:11.009955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:11.009975 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:10:11.010019 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:11.010023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:11.010083 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:11.010190 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:11.011818 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.012763 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.014322 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.014967 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.015073 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:11.018554 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:11.018561 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:11.018567 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:10:11.018624 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:11.021340 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:11.022091 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:11.022097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:11.022099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:11.022236 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:11.022793 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:11.022799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:11.022802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:11.023160 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:11.023428 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:11.023763 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 03:10:11.024020 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":472,"threads_started":1}
I20260926 03:10:11.024801 16941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:11.003977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.3:52837)
0926 03:10:11.004758 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:11.004800 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:11.006600 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:11.007347 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:11.007356 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:11.007962 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:10:11.008681 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:11.008711 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.010237 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.010243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:11.011720 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:11.011740 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.012645 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.012653 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:11.012955 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:11.013623 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:10:11.013651 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:10:11.016402 (+ 2751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:10:11.020262 (+ 3860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:10:11.020271 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:10:11.020287 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:10:11.020767 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:10:11.021163 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:10:11.021167 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:10:11.021170 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:10:11.021384 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:10:11.021956 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:10:11.021966 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:10:11.022425 (+ 459us) client_negotiation.cc:769] Sending connection context
0926 03:10:11.022793 (+ 368us) client_negotiation.cc:240] Negotiation successful
0926 03:10:11.023263 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":320,"threads_started":1}
I20260926 03:10:11.026394 16939 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39617
I20260926 03:10:11.026729 16939 heartbeater.cc:464] Registering TS with master...
I20260926 03:10:11.027482 16939 heartbeater.cc:511] Master 127.0.97.62:39617 requested a full tablet report, sending...
I20260926 03:10:11.029062 16491 ts_manager.cc:194] Registered new tserver with Master: 42c7171b04bf4c4d9650ef6e041aa932 (127.0.97.3:43637)
I20260926 03:10:11.029870 16491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52837
I20260926 03:10:11.039721 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:10:11.046813 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:11.040722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44010 (local address 127.0.97.62:39617)
0926 03:10:11.040912 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:11.040917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:11.041365 (+ 448us) server_negotiation.cc:407] Connection header received
0926 03:10:11.041548 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:11.041552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:11.041742 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:11.041879 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:10:11.042325 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.043199 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.044834 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.045310 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.045418 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:11.045997 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:10:11.046229 (+ 232us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 03:10:11.046600 (+ 371us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44010: token expired
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 03:10:11.046936 16943 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44010: token expired
W20260926 03:10:11.047555 16413 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:39617)
I20260926 03:10:11.061909 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:11.048017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44014 (local address 127.0.97.62:39617)
0926 03:10:11.048209 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:11.048215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:11.048234 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:10:11.048356 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:11.048361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:11.048427 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:11.048507 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:11.049865 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.050652 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.052044 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.052457 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.052564 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:11.055229 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:11.055234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:11.055238 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:10:11.055272 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:11.057939 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:11.058894 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:11.058900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:11.058903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:11.059085 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:11.059596 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:11.059602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:11.059605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:11.060021 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:11.060241 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:11.061355 (+ 1114us) server_negotiation.cc:299] Negotiation successful
0926 03:10:11.061591 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 03:10:11.065860 16491 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:10:11.073616 16491 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:10:11.098762 16676 heartbeater.cc:503] Master 127.0.97.62:39617 was elected leader, sending a full tablet report...
I20260926 03:10:11.104377 16949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:11.085461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43637 (local address 127.0.0.1:45088)
0926 03:10:11.091616 (+ 6155us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:11.091633 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:11.091737 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:11.093130 (+ 1393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:11.093140 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:11.093177 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:10:11.093683 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:11.093697 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.096070 (+ 2373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.096077 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:11.099610 (+ 3533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:11.099630 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.103202 (+ 3572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.103210 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:11.103482 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:11.103540 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 03:10:11.103686 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 03:10:11.103917 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6041,"thread_start_us":4874,"threads_started":1}
I20260926 03:10:11.105446 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:11.086093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45088 (local address 127.0.97.3:43637)
0926 03:10:11.092066 (+ 5973us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:11.092077 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:11.092118 (+ 41us) server_negotiation.cc:407] Connection header received
0926 03:10:11.092229 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:11.092238 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:11.092542 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:11.092761 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:10:11.093960 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.095871 (+ 1911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.099891 (+ 4020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.102574 (+ 2683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.102724 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:11.102995 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:11.104822 (+ 1827us) server_negotiation.cc:299] Negotiation successful
0926 03:10:11.105048 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5771,"thread_start_us":5605,"threads_started":1}
I20260926 03:10:11.106057 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:11.085483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58368 (local address 127.0.97.1:44023)
0926 03:10:11.092326 (+ 6843us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:11.092335 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:11.092361 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:10:11.092453 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:11.092461 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:11.094186 (+ 1725us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:11.094416 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:10:11.095433 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.098767 (+ 3334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.101412 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.103840 (+ 2428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.103976 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:11.104237 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:11.105502 (+ 1265us) server_negotiation.cc:299] Negotiation successful
0926 03:10:11.105716 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6689,"thread_start_us":6531,"threads_started":1}
I20260926 03:10:11.105353 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:11.085276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44023 (local address 127.0.0.1:58368)
0926 03:10:11.090799 (+ 5523us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:11.090861 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:11.091055 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:11.094747 (+ 3692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:11.094753 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:11.094787 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:10:11.095218 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:11.095229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.098978 (+ 3749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.098985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:11.101194 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:11.101213 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.104410 (+ 3197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.104417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:11.104630 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:11.104670 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 03:10:11.104762 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 03:10:11.104930 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5194,"thread_start_us":6073,"threads_started":1}
I20260926 03:10:11.110477 16611 tablet_service.cc:1467] Processing CreateTablet for tablet a4ccedce23b249719e11bd5a7d7aa4a2 (DEFAULT_TABLE table=test-table [id=e45444aabb724318968e3975aa095076]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:10:11.110890 16874 tablet_service.cc:1467] Processing CreateTablet for tablet a4ccedce23b249719e11bd5a7d7aa4a2 (DEFAULT_TABLE table=test-table [id=e45444aabb724318968e3975aa095076]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:10:11.112016 16611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4ccedce23b249719e11bd5a7d7aa4a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:10:11.112303 16874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4ccedce23b249719e11bd5a7d7aa4a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:10:11.114149 16949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:11.092365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35919 (local address 127.0.0.1:48220)
0926 03:10:11.105936 (+ 13571us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:11.105961 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:11.106086 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:11.107534 (+ 1448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:11.107540 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:11.107565 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:10:11.107760 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:11.107772 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.109539 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.109544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:11.111555 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:11.111575 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.113362 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.113368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:11.113551 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:11.113587 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 03:10:11.113696 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 03:10:11.113865 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13449,"thread_start_us":14564,"threads_started":1}
I20260926 03:10:11.114238 16953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:11.091719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48220 (local address 127.0.97.2:35919)
0926 03:10:11.106889 (+ 15170us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:11.106900 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:11.106934 (+ 34us) server_negotiation.cc:407] Connection header received
0926 03:10:11.107036 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:11.107043 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:11.107341 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:11.107527 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:10:11.107897 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.109367 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.111753 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.113160 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.113312 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:11.113505 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:11.113758 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 03:10:11.113927 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14985,"thread_start_us":14819,"threads_started":1}
I20260926 03:10:11.118888 16743 tablet_service.cc:1467] Processing CreateTablet for tablet a4ccedce23b249719e11bd5a7d7aa4a2 (DEFAULT_TABLE table=test-table [id=e45444aabb724318968e3975aa095076]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:10:11.120420 16743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4ccedce23b249719e11bd5a7d7aa4a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:10:11.124750 16955 tablet_bootstrap.cc:492] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac: Bootstrap starting.
I20260926 03:10:11.128943 16955 tablet_bootstrap.cc:654] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac: Neither blocks nor log segments found. Creating new log.
I20260926 03:10:11.129905 16956 tablet_bootstrap.cc:492] T a4ccedce23b249719e11bd5a7d7aa4a2 P 42c7171b04bf4c4d9650ef6e041aa932: Bootstrap starting.
I20260926 03:10:11.130276 16955 log.cc:824] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac: Log is configured to *not* fsync() on all Append() calls
I20260926 03:10:11.133409 16956 tablet_bootstrap.cc:654] T a4ccedce23b249719e11bd5a7d7aa4a2 P 42c7171b04bf4c4d9650ef6e041aa932: Neither blocks nor log segments found. Creating new log.
I20260926 03:10:11.133558 16955 tablet_bootstrap.cc:492] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac: No bootstrap required, opened a new log
I20260926 03:10:11.133867 16955 ts_tablet_manager.cc:1397] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260926 03:10:11.134719 16956 log.cc:824] T a4ccedce23b249719e11bd5a7d7aa4a2 P 42c7171b04bf4c4d9650ef6e041aa932: Log is configured to *not* fsync() on all Append() calls
I20260926 03:10:11.135210 16958 tablet_bootstrap.cc:492] T a4ccedce23b249719e11bd5a7d7aa4a2 P ff233c21ff6f4757b49bfe0ad0c65e33: Bootstrap starting.
I20260926 03:10:11.138753 16958 tablet_bootstrap.cc:654] T a4ccedce23b249719e11bd5a7d7aa4a2 P ff233c21ff6f4757b49bfe0ad0c65e33: Neither blocks nor log segments found. Creating new log.
I20260926 03:10:11.138913 16956 tablet_bootstrap.cc:492] T a4ccedce23b249719e11bd5a7d7aa4a2 P 42c7171b04bf4c4d9650ef6e041aa932: No bootstrap required, opened a new log
I20260926 03:10:11.139235 16956 ts_tablet_manager.cc:1397] T a4ccedce23b249719e11bd5a7d7aa4a2 P 42c7171b04bf4c4d9650ef6e041aa932: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260926 03:10:11.139918 16958 log.cc:824] T a4ccedce23b249719e11bd5a7d7aa4a2 P ff233c21ff6f4757b49bfe0ad0c65e33: Log is configured to *not* fsync() on all Append() calls
I20260926 03:10:11.142547 16958 tablet_bootstrap.cc:492] T a4ccedce23b249719e11bd5a7d7aa4a2 P ff233c21ff6f4757b49bfe0ad0c65e33: No bootstrap required, opened a new log
I20260926 03:10:11.142812 16958 ts_tablet_manager.cc:1397] T a4ccedce23b249719e11bd5a7d7aa4a2 P ff233c21ff6f4757b49bfe0ad0c65e33: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260926 03:10:11.144258 16955 raft_consensus.cc:348] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67353804c3504e4287cd5aa5efe6e8ac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44023 } } peers { permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } peers { permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 } }
I20260926 03:10:11.144742 16955 raft_consensus.cc:374] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:10:11.144855 16955 raft_consensus.cc:729] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67353804c3504e4287cd5aa5efe6e8ac, State: Initialized, Role: FOLLOWER
I20260926 03:10:11.145521 16955 consensus_queue.cc:260] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67353804c3504e4287cd5aa5efe6e8ac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44023 } } peers { permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } peers { permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 } }
I20260926 03:10:11.147418 16955 ts_tablet_manager.cc:1428] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260926 03:10:11.149227 16956 raft_consensus.cc:348] T a4ccedce23b249719e11bd5a7d7aa4a2 P 42c7171b04bf4c4d9650ef6e041aa932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67353804c3504e4287cd5aa5efe6e8ac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44023 } } peers { permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } peers { permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 } }
I20260926 03:10:11.149737 16956 raft_consensus.cc:374] T a4ccedce23b249719e11bd5a7d7aa4a2 P 42c7171b04bf4c4d9650ef6e041aa932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:10:11.149842 16956 raft_consensus.cc:729] T a4ccedce23b249719e11bd5a7d7aa4a2 P 42c7171b04bf4c4d9650ef6e041aa932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42c7171b04bf4c4d9650ef6e041aa932, State: Initialized, Role: FOLLOWER
I20260926 03:10:11.150630 16956 consensus_queue.cc:260] T a4ccedce23b249719e11bd5a7d7aa4a2 P 42c7171b04bf4c4d9650ef6e041aa932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67353804c3504e4287cd5aa5efe6e8ac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44023 } } peers { permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } peers { permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 } }
I20260926 03:10:11.151921 16939 heartbeater.cc:503] Master 127.0.97.62:39617 was elected leader, sending a full tablet report...
I20260926 03:10:11.152467 16956 ts_tablet_manager.cc:1428] T a4ccedce23b249719e11bd5a7d7aa4a2 P 42c7171b04bf4c4d9650ef6e041aa932: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260926 03:10:11.152781 16958 raft_consensus.cc:348] T a4ccedce23b249719e11bd5a7d7aa4a2 P ff233c21ff6f4757b49bfe0ad0c65e33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67353804c3504e4287cd5aa5efe6e8ac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44023 } } peers { permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } peers { permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 } }
I20260926 03:10:11.153230 16958 raft_consensus.cc:374] T a4ccedce23b249719e11bd5a7d7aa4a2 P ff233c21ff6f4757b49bfe0ad0c65e33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:10:11.153322 16958 raft_consensus.cc:729] T a4ccedce23b249719e11bd5a7d7aa4a2 P ff233c21ff6f4757b49bfe0ad0c65e33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff233c21ff6f4757b49bfe0ad0c65e33, State: Initialized, Role: FOLLOWER
I20260926 03:10:11.153934 16958 consensus_queue.cc:260] T a4ccedce23b249719e11bd5a7d7aa4a2 P ff233c21ff6f4757b49bfe0ad0c65e33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67353804c3504e4287cd5aa5efe6e8ac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44023 } } peers { permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } peers { permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 } }
I20260926 03:10:11.154352 16961 raft_consensus.cc:482] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:10:11.154642 16961 raft_consensus.cc:504] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67353804c3504e4287cd5aa5efe6e8ac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44023 } } peers { permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } peers { permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 } }
I20260926 03:10:11.156402 16961 leader_election.cc:288] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42c7171b04bf4c4d9650ef6e041aa932 (127.0.97.3:43637), ff233c21ff6f4757b49bfe0ad0c65e33 (127.0.97.2:35919)
I20260926 03:10:11.160562 16808 heartbeater.cc:503] Master 127.0.97.62:39617 was elected leader, sending a full tablet report...
I20260926 03:10:11.161392 16958 ts_tablet_manager.cc:1428] T a4ccedce23b249719e11bd5a7d7aa4a2 P ff233c21ff6f4757b49bfe0ad0c65e33: Time spent starting tablet: real 0.018s user 0.001s sys 0.016s
I20260926 03:10:11.164932 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:11.156732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56365 (local address 127.0.97.3:43637)
0926 03:10:11.156951 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:11.156958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:11.157670 (+ 712us) server_negotiation.cc:407] Connection header received
0926 03:10:11.157959 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:11.157965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:11.158044 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:11.158107 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:10:11.159238 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.160386 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.162780 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.163981 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.164104 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:11.164206 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:11.164552 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 03:10:11.164690 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:10:11.164983 16964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:11.156733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43637 (local address 127.0.97.1:56365)
0926 03:10:11.157584 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:11.157603 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:11.157776 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:11.158587 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:11.158593 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:11.158624 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:10:11.159012 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:11.159025 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.160563 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.160569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:11.162622 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:11.162635 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.164206 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.164213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:11.164387 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:11.164424 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 03:10:11.164505 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 03:10:11.164649 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":545,"threads_started":1}
I20260926 03:10:11.166296 16953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:11.158027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48445 (local address 127.0.97.2:35919)
0926 03:10:11.158442 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:11.158450 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:11.159688 (+ 1238us) server_negotiation.cc:407] Connection header received
0926 03:10:11.159910 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:11.159917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:11.160001 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:11.160094 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:10:11.160890 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.161936 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.164027 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.165181 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.165327 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:11.165417 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:11.165923 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 03:10:11.166059 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213}
I20260926 03:10:11.166355 16965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:11.157736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35919 (local address 127.0.97.1:48445)
0926 03:10:11.159605 (+ 1869us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:11.159620 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:11.159774 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:11.160245 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:11.160249 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:11.160272 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:10:11.160692 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:11.160705 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.162072 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.162078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:11.163865 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:11.163879 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.165570 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.165576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:11.165746 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:11.165779 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:10:11.165871 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 03:10:11.166014 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1754,"thread_start_us":1597,"threads_started":1}
I20260926 03:10:11.166177 16894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4ccedce23b249719e11bd5a7d7aa4a2" candidate_uuid: "67353804c3504e4287cd5aa5efe6e8ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42c7171b04bf4c4d9650ef6e041aa932" is_pre_election: true
I20260926 03:10:11.166764 16894 raft_consensus.cc:2454] T a4ccedce23b249719e11bd5a7d7aa4a2 P 42c7171b04bf4c4d9650ef6e041aa932 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67353804c3504e4287cd5aa5efe6e8ac in term 0.
I20260926 03:10:11.167543 16566 leader_election.cc:302] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42c7171b04bf4c4d9650ef6e041aa932, 67353804c3504e4287cd5aa5efe6e8ac; no voters:
I20260926 03:10:11.168076 16763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4ccedce23b249719e11bd5a7d7aa4a2" candidate_uuid: "67353804c3504e4287cd5aa5efe6e8ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" is_pre_election: true
I20260926 03:10:11.168206 16961 raft_consensus.cc:2781] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:10:11.168363 16961 raft_consensus.cc:482] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:10:11.168467 16961 raft_consensus.cc:3037] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:10:11.168579 16763 raft_consensus.cc:2454] T a4ccedce23b249719e11bd5a7d7aa4a2 P ff233c21ff6f4757b49bfe0ad0c65e33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67353804c3504e4287cd5aa5efe6e8ac in term 0.
I20260926 03:10:11.170958 16961 raft_consensus.cc:504] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67353804c3504e4287cd5aa5efe6e8ac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44023 } } peers { permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } peers { permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 } }
I20260926 03:10:11.171607 16961 leader_election.cc:288] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 1 election: Requested vote from peers 42c7171b04bf4c4d9650ef6e041aa932 (127.0.97.3:43637), ff233c21ff6f4757b49bfe0ad0c65e33 (127.0.97.2:35919)
I20260926 03:10:11.172087 16894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4ccedce23b249719e11bd5a7d7aa4a2" candidate_uuid: "67353804c3504e4287cd5aa5efe6e8ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42c7171b04bf4c4d9650ef6e041aa932"
I20260926 03:10:11.172134 16763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4ccedce23b249719e11bd5a7d7aa4a2" candidate_uuid: "67353804c3504e4287cd5aa5efe6e8ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33"
I20260926 03:10:11.172318 16763 raft_consensus.cc:3037] T a4ccedce23b249719e11bd5a7d7aa4a2 P ff233c21ff6f4757b49bfe0ad0c65e33 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:10:11.172310 16894 raft_consensus.cc:3037] T a4ccedce23b249719e11bd5a7d7aa4a2 P 42c7171b04bf4c4d9650ef6e041aa932 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:10:11.174618 16763 raft_consensus.cc:2454] T a4ccedce23b249719e11bd5a7d7aa4a2 P ff233c21ff6f4757b49bfe0ad0c65e33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67353804c3504e4287cd5aa5efe6e8ac in term 1.
I20260926 03:10:11.174733 16894 raft_consensus.cc:2454] T a4ccedce23b249719e11bd5a7d7aa4a2 P 42c7171b04bf4c4d9650ef6e041aa932 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67353804c3504e4287cd5aa5efe6e8ac in term 1.
I20260926 03:10:11.175053 16566 leader_election.cc:302] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67353804c3504e4287cd5aa5efe6e8ac, ff233c21ff6f4757b49bfe0ad0c65e33; no voters:
I20260926 03:10:11.175374 16961 raft_consensus.cc:2781] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:10:11.175865 16961 raft_consensus.cc:686] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 1 LEADER]: Becoming Leader. State: Replica: 67353804c3504e4287cd5aa5efe6e8ac, State: Running, Role: LEADER
I20260926 03:10:11.176349 16961 consensus_queue.cc:237] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67353804c3504e4287cd5aa5efe6e8ac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44023 } } peers { permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } peers { permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 } }
I20260926 03:10:11.181704 16491 catalog_manager.cc:5495] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 67353804c3504e4287cd5aa5efe6e8ac (127.0.97.1). New cstate: current_term: 1 leader_uuid: "67353804c3504e4287cd5aa5efe6e8ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67353804c3504e4287cd5aa5efe6e8ac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 } health_report { overall_health: UNKNOWN } } }
I20260926 03:10:11.198907 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:11.193559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58376 (local address 127.0.97.1:44023)
0926 03:10:11.193823 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:11.193829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:11.193845 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:10:11.193930 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:11.193934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:11.194008 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:11.194134 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:10:11.194544 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.195301 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.196700 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.197111 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.197223 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:11.197771 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:10:11.198281 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:10:11.198419 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:11.198558 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 03:10:11.198649 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:10:11.207201 16894 raft_consensus.cc:1261] T a4ccedce23b249719e11bd5a7d7aa4a2 P 42c7171b04bf4c4d9650ef6e041aa932 [term 1 FOLLOWER]: Refusing update from remote peer 67353804c3504e4287cd5aa5efe6e8ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:10:11.207231 16763 raft_consensus.cc:1261] T a4ccedce23b249719e11bd5a7d7aa4a2 P ff233c21ff6f4757b49bfe0ad0c65e33 [term 1 FOLLOWER]: Refusing update from remote peer 67353804c3504e4287cd5aa5efe6e8ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:10:11.208161 16966 consensus_queue.cc:1035] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:10:11.208525 16961 consensus_queue.cc:1035] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:10:11.221680 16970 mvcc.cc:205] Tried to move back new op lower bound from 7333446497094668288 to 7333446496982675456. Current Snapshot: MvccSnapshot[applied={T|T < 7333446497094668288}]
I20260926 03:10:11.237087 16491 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:10:11.237372 16491 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:10:11.241133 16491 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54: Sending DeleteTablet for 3 replicas of tablet a4ccedce23b249719e11bd5a7d7aa4a2
I20260926 03:10:11.242398 16611 tablet_service.cc:1514] Processing DeleteTablet for tablet a4ccedce23b249719e11bd5a7d7aa4a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:10:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58368
I20260926 03:10:11.242535 16743 tablet_service.cc:1514] Processing DeleteTablet for tablet a4ccedce23b249719e11bd5a7d7aa4a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:10:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48220
I20260926 03:10:11.242659 16874 tablet_service.cc:1514] Processing DeleteTablet for tablet a4ccedce23b249719e11bd5a7d7aa4a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:10:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45088
I20260926 03:10:11.243685 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 16528
I20260926 03:10:11.258658 16979 tablet_replica.cc:330] T a4ccedce23b249719e11bd5a7d7aa4a2 P ff233c21ff6f4757b49bfe0ad0c65e33: stopping tablet replica
I20260926 03:10:11.259433 16979 raft_consensus.cc:2229] T a4ccedce23b249719e11bd5a7d7aa4a2 P ff233c21ff6f4757b49bfe0ad0c65e33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:10:11.260955 16979 raft_consensus.cc:2258] T a4ccedce23b249719e11bd5a7d7aa4a2 P ff233c21ff6f4757b49bfe0ad0c65e33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:10:11.263834 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:11.244501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58388 (local address 127.0.97.1:44023)
0926 03:10:11.244704 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:11.244711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:11.244759 (+ 48us) server_negotiation.cc:407] Connection header received
0926 03:10:11.244954 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:11.244960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:11.245048 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:11.245134 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:11.246994 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.248340 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.250335 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.251131 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.251258 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:11.254896 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:11.254921 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:11.254948 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:10:11.255033 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:11.260019 (+ 4986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:11.260952 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:11.260962 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:11.260966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:11.261207 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:11.261632 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:11.261637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:11.261641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:11.262155 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:11.262488 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:11.263064 (+ 576us) server_negotiation.cc:299] Negotiation successful
0926 03:10:11.263375 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 03:10:11.264569 16671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:10:11.265667 16979 ts_tablet_manager.cc:1905] T a4ccedce23b249719e11bd5a7d7aa4a2 P ff233c21ff6f4757b49bfe0ad0c65e33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:10:11.266777 16980 tablet_replica.cc:330] T a4ccedce23b249719e11bd5a7d7aa4a2 P 42c7171b04bf4c4d9650ef6e041aa932: stopping tablet replica
I20260926 03:10:11.267871 16980 raft_consensus.cc:2229] T a4ccedce23b249719e11bd5a7d7aa4a2 P 42c7171b04bf4c4d9650ef6e041aa932 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:10:11.268548 16980 raft_consensus.cc:2258] T a4ccedce23b249719e11bd5a7d7aa4a2 P 42c7171b04bf4c4d9650ef6e041aa932 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:10:11.271132 16980 ts_tablet_manager.cc:1905] T a4ccedce23b249719e11bd5a7d7aa4a2 P 42c7171b04bf4c4d9650ef6e041aa932: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:10:11.274253 16979 ts_tablet_manager.cc:1918] T a4ccedce23b249719e11bd5a7d7aa4a2 P ff233c21ff6f4757b49bfe0ad0c65e33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:10:11.274421 16979 log.cc:1196] T a4ccedce23b249719e11bd5a7d7aa4a2 P ff233c21ff6f4757b49bfe0ad0c65e33: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/wal/wals/a4ccedce23b249719e11bd5a7d7aa4a2
I20260926 03:10:11.274935 16979 ts_tablet_manager.cc:1939] T a4ccedce23b249719e11bd5a7d7aa4a2 P ff233c21ff6f4757b49bfe0ad0c65e33: Deleting consensus metadata
I20260926 03:10:11.276582 16477 catalog_manager.cc:4841] TS ff233c21ff6f4757b49bfe0ad0c65e33 (127.0.97.2:35919): tablet a4ccedce23b249719e11bd5a7d7aa4a2 (table test-table [id=e45444aabb724318968e3975aa095076]) successfully deleted
I20260926 03:10:11.278801 16980 ts_tablet_manager.cc:1918] T a4ccedce23b249719e11bd5a7d7aa4a2 P 42c7171b04bf4c4d9650ef6e041aa932: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:10:11.278982 16980 log.cc:1196] T a4ccedce23b249719e11bd5a7d7aa4a2 P 42c7171b04bf4c4d9650ef6e041aa932: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/wal/wals/a4ccedce23b249719e11bd5a7d7aa4a2
I20260926 03:10:11.279513 16980 ts_tablet_manager.cc:1939] T a4ccedce23b249719e11bd5a7d7aa4a2 P 42c7171b04bf4c4d9650ef6e041aa932: Deleting consensus metadata
I20260926 03:10:11.281488 16477 catalog_manager.cc:4841] TS 42c7171b04bf4c4d9650ef6e041aa932 (127.0.97.3:43637): tablet a4ccedce23b249719e11bd5a7d7aa4a2 (table test-table [id=e45444aabb724318968e3975aa095076]) successfully deleted
I20260926 03:10:11.368201 16978 tablet_replica.cc:330] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac: stopping tablet replica
I20260926 03:10:11.368957 16978 raft_consensus.cc:2229] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:10:11.369052 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 16528
I20260926 03:10:11.384346 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 16680
W20260926 03:10:11.384356 16476 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44023 (error 108)
I20260926 03:10:11.386377 16949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:11.385888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44023 (local address 127.0.0.1:58400)
0926 03:10:11.386072 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:11.386183 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 03:10:11.386649 16476 catalog_manager.cc:4584] TS 67353804c3504e4287cd5aa5efe6e8ac (127.0.97.1:44023): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a4ccedce23b249719e11bd5a7d7aa4a2: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44023: connect: Connection refused (error 111)
I20260926 03:10:11.399098 16953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:11.385106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48230 (local address 127.0.97.2:35919)
0926 03:10:11.385287 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:11.385293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:11.385474 (+ 181us) server_negotiation.cc:407] Connection header received
0926 03:10:11.385599 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:11.385604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:11.385694 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:11.385876 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:11.387430 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.388143 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.389418 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.389867 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.389962 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:11.392919 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:11.392933 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:11.392949 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:10:11.392988 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:11.395811 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:11.396655 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:11.396663 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:11.396666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:11.396867 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:11.397280 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:11.397285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:11.397287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:11.397821 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:11.398066 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:11.398408 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 03:10:11.398750 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:10:11.399739 16803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:10:11.440356 16949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:11.439768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44023 (local address 127.0.0.1:58404)
0926 03:10:11.440044 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:11.440173 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 03:10:11.441563 16949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:11.441108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44023 (local address 127.0.0.1:58410)
0926 03:10:11.441271 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:11.441372 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 03:10:11.490450 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 16680
I20260926 03:10:11.502382 16949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:11.501854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44023 (local address 127.0.0.1:58426)
0926 03:10:11.502098 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:11.502215 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 03:10:11.503614 16949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:11.503162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44023 (local address 127.0.0.1:58436)
0926 03:10:11.503363 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:11.503441 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 03:10:11.505246 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 16811
I20260926 03:10:11.520833 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:11.505982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45094 (local address 127.0.97.3:43637)
0926 03:10:11.506180 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:11.506186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:11.506208 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:10:11.506299 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:11.506305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:11.506396 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:11.506518 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:11.507817 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.508960 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.510325 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:11.510983 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:11.511107 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:11.513839 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:11.513859 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:11.513880 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:10:11.513932 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:11.517660 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:11.518582 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:11.518591 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:11.518594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:11.518791 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:11.519194 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:11.519199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:11.519203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:11.519671 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:11.519971 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:11.520252 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 03:10:11.520491 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:10:11.521464 16934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:10:11.584481 16949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:11.583871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44023 (local address 127.0.0.1:58444)
0926 03:10:11.584131 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:11.584308 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260926 03:10:11.586293 16949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:11.585775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44023 (local address 127.0.0.1:58458)
0926 03:10:11.586002 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:11.586114 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260926 03:10:11.612062 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 16811
I20260926 03:10:11.625674 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 16458
I20260926 03:10:11.626959 16521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:10:11.739859 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 16458
I20260926 03:10:31.759681 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39617
--webserver_interface=127.0.97.62
--webserver_port=43785
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:10:32.114053 16986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:10:32.114418 16986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:10:32.114570 16986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:10:32.122217 16986 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 03:10:32.122331 16986 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 03:10:32.122367 16986 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 03:10:32.122428 16986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:10:32.122531 16986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:10:32.122622 16986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:10:32.122677 16986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:10:32.122754 16986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:10:32.133445 16986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=43785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:10:32.134932 16986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:10:32.136390 16986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:10:32.148420 16991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:32.148504 16994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:32.148475 16992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:10:32.149128 16986 server_base.cc:1034] running on GCE node
I20260926 03:10:32.149968 16986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:10:32.151532 16986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:10:32.152762 16986 hybrid_clock.cc:648] HybridClock initialized: now 1790392232152703 us; error 38 us; skew 500 ppm
I20260926 03:10:32.153120 16986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:10:32 dist-test-slave-hmbw krb5kdc[15906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:10:32.156358 16986 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:10:32.157943 16986 webserver.cc:460] Webserver started at http://127.0.97.62:43785/ using document root <none> and password file <none>
I20260926 03:10:32.158725 16986 fs_manager.cc:359] Metadata directory not provided
I20260926 03:10:32.158823 16986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:10:32.164582 16986 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 03:10:32.167732 17001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:32.169234 16986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:10:32.169422 16986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "cf655247a70b4c97bf4249d8e522ce54"
format_stamp: "Formatted at 2026-09-26 03:09:46 on dist-test-slave-hmbw"
server_key: "d61fd23f794e99eba896c04d9f38e2d9"
server_key_iv: "0c397aa115c0b2889dc33b34f23b0762"
server_key_version: "encryptionkey@0"
I20260926 03:10:32.170467 16986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:10:32.209824 16986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:10:32.214448 16986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:10:32.214756 16986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:10:32.232630 16986 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39617
I20260926 03:10:32.232661 17052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39617 every 8 connection(s)
I20260926 03:10:32.234251 16986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:10:32.238545 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 16986
I20260926 03:10:32.239396 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:44023
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:39617
--webserver_port=36391
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:10:32.241757 17053 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:10:32.245430 17053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54: Bootstrap starting.
I20260926 03:10:32.275079 17053 log.cc:824] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54: Log is configured to *not* fsync() on all Append() calls
I20260926 03:10:32.295652 17053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:10:32.296214 17053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54: Bootstrap complete.
I20260926 03:10:32.305295 17053 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf655247a70b4c97bf4249d8e522ce54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39617 } }
I20260926 03:10:32.306265 17053 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf655247a70b4c97bf4249d8e522ce54, State: Initialized, Role: FOLLOWER
I20260926 03:10:32.307044 17053 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf655247a70b4c97bf4249d8e522ce54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39617 } }
I20260926 03:10:32.307286 17053 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:10:32.307400 17053 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:10:32.307578 17053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [term 2 FOLLOWER]: Advancing to term 3
I20260926 03:10:32.312014 17053 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf655247a70b4c97bf4249d8e522ce54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39617 } }
I20260926 03:10:32.312605 17053 leader_election.cc:302] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf655247a70b4c97bf4249d8e522ce54; no voters:
I20260926 03:10:32.313297 17053 leader_election.cc:288] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 03:10:32.313532 17057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 03:10:32.314662 17057 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [term 3 LEADER]: Becoming Leader. State: Replica: cf655247a70b4c97bf4249d8e522ce54, State: Running, Role: LEADER
I20260926 03:10:32.315330 17057 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf655247a70b4c97bf4249d8e522ce54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39617 } }
I20260926 03:10:32.315586 17053 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:10:32.318519 17059 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf655247a70b4c97bf4249d8e522ce54. Latest consensus state: current_term: 3 leader_uuid: "cf655247a70b4c97bf4249d8e522ce54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf655247a70b4c97bf4249d8e522ce54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39617 } } }
I20260926 03:10:32.319070 17059 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [sys.catalog]: This master's current role is: LEADER
I20260926 03:10:32.318912 17058 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cf655247a70b4c97bf4249d8e522ce54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf655247a70b4c97bf4249d8e522ce54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39617 } } }
I20260926 03:10:32.319238 17058 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54 [sys.catalog]: This master's current role is: LEADER
I20260926 03:10:32.322257 17066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:10:32.332530 17066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:10:32.334210 17066 catalog_manager.cc:1259] Loaded cluster ID: d6a6ca50235840d78b926b99984f1578
I20260926 03:10:32.334343 17066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:10:32.338915 17066 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:10:32.340833 17066 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54: Loaded TSK: 0
I20260926 03:10:32.341007 17066 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cf655247a70b4c97bf4249d8e522ce54: Loaded TSK: 1
F20260926 03:10:32.342370 17066 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7f767609ad google::LogMessage::Fail() at ??:0
@ 0x7f7f7676388a google::LogMessage::SendToLog() at ??:0
@ 0x7f7f767604de google::LogMessage::Flush() at ??:0
@ 0x7f7f76761a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7f7a4c47d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f7f7a4b373e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f7f8c18f1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f7f8c18e783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f7f8c21a2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f7f8c21a0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f7f8c2645e9 std::function<>::operator()() at ??:0
@ 0x7f7f8c192cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f7f8c190c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7f8c218610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f7f8c218439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7f793ac345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7f793af91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f7f793af799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7f7938aeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7f7c6646db start_thread at ??:0
@ 0x7f7f755fa61f clone at ??:0
W20260926 03:10:32.673566 17055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:10:32.673856 17055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:10:32.673949 17055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:10:32.681512 17055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:10:32.681628 17055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:10:32.681751 17055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:10:32.691857 17055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:44023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36391
--tserver_master_addrs=127.0.97.62:39617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:10:32.693114 17055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:10:32.694378 17055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:10:32.706951 17100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:32.707151 17103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:32.706980 17101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:10:32.708500 17055 server_base.cc:1034] running on GCE node
I20260926 03:10:32.709177 17055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:10:32.710160 17055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:10:32.711383 17055 hybrid_clock.cc:648] HybridClock initialized: now 1790392232711335 us; error 35 us; skew 500 ppm
I20260926 03:10:32.711762 17055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:10:32 dist-test-slave-hmbw krb5kdc[15906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:10:32.714984 17055 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:10:32.716593 17055 webserver.cc:460] Webserver started at http://127.0.97.1:36391/ using document root <none> and password file <none>
I20260926 03:10:32.717283 17055 fs_manager.cc:359] Metadata directory not provided
I20260926 03:10:32.717388 17055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:10:32.723100 17055 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 03:10:32.726199 17110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:32.727666 17055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:10:32.727856 17055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "67353804c3504e4287cd5aa5efe6e8ac"
format_stamp: "Formatted at 2026-09-26 03:09:47 on dist-test-slave-hmbw"
server_key: "ca21cb183e6400736abd173e7a46aa2f"
server_key_iv: "8ab072d0d9fbd1518980f9f6ed45fe0e"
server_key_version: "encryptionkey@0"
I20260926 03:10:32.728966 17055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:10:32.749573 17055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:10:32.754207 17055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:10:32.754534 17055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:10:32.755594 17055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:10:32.757447 17117 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260926 03:10:32.763324 17055 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 03:10:32.763507 17055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260926 03:10:32.763639 17055 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 03:10:32.767141 17055 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 03:10:32.767243 17055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.003s sys 0.000s
I20260926 03:10:32.767674 17117 tablet_bootstrap.cc:492] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac: Bootstrap starting.
I20260926 03:10:32.793154 17117 log.cc:824] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac: Log is configured to *not* fsync() on all Append() calls
I20260926 03:10:32.808171 17117 tablet_bootstrap.cc:492] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:10:32.808271 17055 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44023
I20260926 03:10:32.808292 17224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44023 every 8 connection(s)
I20260926 03:10:32.808771 17117 tablet_bootstrap.cc:492] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac: Bootstrap complete.
I20260926 03:10:32.809635 17055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:10:32.809989 17117 ts_tablet_manager.cc:1397] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac: Time spent bootstrapping tablet: real 0.043s user 0.026s sys 0.012s
I20260926 03:10:32.815080 17227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:32.812679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.1:55077)
0926 03:10:32.813682 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:32.813958 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":438,"threads_started":1}
W20260926 03:10:32.815855 17225 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39617 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:39617: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
I20260926 03:10:32.816879 17227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:32.816490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.1:43545)
0926 03:10:32.816666 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:32.816747 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 03:10:32.817829 17227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:32.817475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.1:41263)
0926 03:10:32.817627 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:32.817703 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 03:10:32.818114 17225 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 03:10:32.817771 17117 raft_consensus.cc:348] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67353804c3504e4287cd5aa5efe6e8ac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44023 } } peers { permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } peers { permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 } }
I20260926 03:10:32.818884 17117 raft_consensus.cc:729] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67353804c3504e4287cd5aa5efe6e8ac, State: Initialized, Role: FOLLOWER
I20260926 03:10:32.819502 17117 consensus_queue.cc:260] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67353804c3504e4287cd5aa5efe6e8ac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44023 } } peers { permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } peers { permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 } }
I20260926 03:10:32.819873 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 17055
I20260926 03:10:32.820607 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:35919
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:39617
--webserver_port=43237
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:10:32.821115 17227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:32.820805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.1:43159)
0926 03:10:32.820928 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:32.821000 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 03:10:32.821229 17117 ts_tablet_manager.cc:1428] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac: Time spent starting tablet: real 0.011s user 0.014s sys 0.000s
I20260926 03:10:32.822031 17227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:32.821720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.1:60763)
0926 03:10:32.821852 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:32.821914 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 03:10:33.155345 17229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:10:33.155658 17229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:10:33.155762 17229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:10:33.163229 17229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:10:33.163354 17229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:10:33.163481 17229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:10:33.173663 17229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:35919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43237
--tserver_master_addrs=127.0.97.62:39617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:10:33.174984 17229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:10:33.176225 17229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:10:33.189622 17235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:33.189625 17234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:33.189625 17237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:10:33.195411 17229 server_base.cc:1034] running on GCE node
I20260926 03:10:33.196149 17229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:10:33.197176 17229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:10:33.198395 17229 hybrid_clock.cc:648] HybridClock initialized: now 1790392233198340 us; error 45 us; skew 500 ppm
I20260926 03:10:33.198746 17229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:10:33 dist-test-slave-hmbw krb5kdc[15906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:10:33.201740 17229 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:10:33.203423 17229 webserver.cc:460] Webserver started at http://127.0.97.2:43237/ using document root <none> and password file <none>
I20260926 03:10:33.204195 17229 fs_manager.cc:359] Metadata directory not provided
I20260926 03:10:33.204313 17229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:10:33.210021 17229 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 03:10:33.213105 17244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:33.214782 17229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 03:10:33.215063 17229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "ff233c21ff6f4757b49bfe0ad0c65e33"
format_stamp: "Formatted at 2026-09-26 03:09:47 on dist-test-slave-hmbw"
server_key: "28004096e3ce5fc1033a5692df1ba6fc"
server_key_iv: "235d92060848149ef7af1c37619c51b7"
server_key_version: "encryptionkey@0"
I20260926 03:10:33.216152 17229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:10:33.235800 17229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:10:33.240123 17229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:10:33.240389 17229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:10:33.241446 17229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:10:33.242949 17229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:10:33.243052 17229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:33.243167 17229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:10:33.243235 17229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:33.276160 17229 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35919
I20260926 03:10:33.276207 17356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35919 every 8 connection(s)
I20260926 03:10:33.277640 17229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:10:33.282712 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:33.280775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.2:39539)
0926 03:10:33.281522 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:33.281736 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":289,"threads_started":1}
W20260926 03:10:33.283391 17357 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39617 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:39617: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
I20260926 03:10:33.284308 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:33.283943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.2:33135)
0926 03:10:33.284109 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:33.284181 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 03:10:33.285248 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:33.284915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.2:39453)
0926 03:10:33.285057 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:33.285133 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 03:10:33.285463 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 17229
W20260926 03:10:33.285486 17357 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 03:10:33.286019 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:43637
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:39617
--webserver_port=32921
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:10:33.616037 17360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:10:33.616326 17360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:10:33.616421 17360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:10:33.623873 17360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:10:33.623998 17360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:10:33.624116 17360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:10:33.634227 17360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:43637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=32921
--tserver_master_addrs=127.0.97.62:39617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:10:33.635540 17360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:10:33.636807 17360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:10:33.649893 17368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:33.650089 17365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:33.650064 17366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:10:33.651214 17360 server_base.cc:1034] running on GCE node
I20260926 03:10:33.651906 17360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:10:33.652937 17360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:10:33.654167 17360 hybrid_clock.cc:648] HybridClock initialized: now 1790392233654102 us; error 54 us; skew 500 ppm
I20260926 03:10:33.654644 17360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:10:33 dist-test-slave-hmbw krb5kdc[15906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:10:33.658092 17360 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:10:33.659812 17360 webserver.cc:460] Webserver started at http://127.0.97.3:32921/ using document root <none> and password file <none>
I20260926 03:10:33.660523 17360 fs_manager.cc:359] Metadata directory not provided
I20260926 03:10:33.660645 17360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:10:33.666381 17360 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:10:33.669440 17375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:33.671119 17360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 03:10:33.671295 17360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "42c7171b04bf4c4d9650ef6e041aa932"
format_stamp: "Formatted at 2026-09-26 03:09:48 on dist-test-slave-hmbw"
server_key: "9ed07e54a54a972ccb99f52d5bd1ee54"
server_key_iv: "76ee9f2deb7209212432962d1354bd38"
server_key_version: "encryptionkey@0"
I20260926 03:10:33.672351 17360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:10:33.693758 17360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:10:33.698163 17360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:10:33.698446 17360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:10:33.699594 17360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:10:33.701088 17360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:10:33.701185 17360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:33.701299 17360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:10:33.701370 17360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:33.735175 17360 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43637
I20260926 03:10:33.735230 17487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43637 every 8 connection(s)
I20260926 03:10:33.736663 17360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:10:33.739542 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 17360
I20260926 03:10:33.741786 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:33.739824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.3:37643)
0926 03:10:33.740621 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:33.740890 (+ 269us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":318,"threads_started":1}
W20260926 03:10:33.742383 17488 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39617 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:39617: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
I20260926 03:10:33.743193 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:33.742869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.3:45237)
0926 03:10:33.743014 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:33.743084 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 03:10:33.743968 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:33.743654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.3:33417)
0926 03:10:33.743787 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:33.743858 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 03:10:33.744201 17488 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 03:10:33.823766 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:33.822931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.1:47011)
0926 03:10:33.823462 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:33.823558 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":261,"threads_started":1}
I20260926 03:10:34.286896 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:34.286061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.2:39689)
0926 03:10:34.286540 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:34.286634 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":204,"threads_started":1}
I20260926 03:10:34.606756 17494 raft_consensus.cc:482] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:10:34.607002 17494 raft_consensus.cc:504] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67353804c3504e4287cd5aa5efe6e8ac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44023 } } peers { permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } peers { permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 } }
I20260926 03:10:34.608520 17494 leader_election.cc:288] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42c7171b04bf4c4d9650ef6e041aa932 (127.0.97.3:43637), ff233c21ff6f4757b49bfe0ad0c65e33 (127.0.97.2:35919)
Sep 26 03:10:34 dist-test-slave-hmbw krb5kdc[15906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
Sep 26 03:10:34 dist-test-slave-hmbw krb5kdc[15906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:10:34.632625 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:34.609074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40707 (local address 127.0.97.3:43637)
0926 03:10:34.609634 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:34.609648 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:34.610191 (+ 543us) server_negotiation.cc:407] Connection header received
0926 03:10:34.613154 (+ 2963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:34.613166 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:34.613667 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:34.614013 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:34.616657 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:34.618589 (+ 1932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:34.620770 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:34.621490 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:34.621667 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:34.625910 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:34.625920 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:34.625935 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:10:34.625980 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:34.628799 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:34.629618 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:34.629626 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:34.629631 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:34.629794 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:34.630247 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:34.630251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:34.630253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:34.630824 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:34.630999 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:34.631809 (+ 810us) server_negotiation.cc:299] Negotiation successful
0926 03:10:34.632209 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":291,"threads_started":1}
I20260926 03:10:34.632634 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:34.610651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60087 (local address 127.0.97.2:35919)
0926 03:10:34.611384 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:34.611397 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:34.611438 (+ 41us) server_negotiation.cc:407] Connection header received
0926 03:10:34.613486 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:34.613496 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:34.613850 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:34.614145 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:34.616657 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:34.618591 (+ 1934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:34.620771 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:34.621453 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:34.621598 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:34.625794 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:34.625808 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:34.625825 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:10:34.625875 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:34.628739 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:34.629625 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:34.629633 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:34.629639 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:34.629795 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:34.630257 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:34.630260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:34.630262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:34.630830 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:34.631008 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:34.631800 (+ 792us) server_negotiation.cc:299] Negotiation successful
0926 03:10:34.632266 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":300,"threads_started":1}
I20260926 03:10:34.632704 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:34.608818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43637 (local address 127.0.97.1:40707)
0926 03:10:34.609970 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:34.610017 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:34.610715 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:34.614149 (+ 3434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:34.614161 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:34.615274 (+ 1113us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:10:34.616408 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:34.616437 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:34.618807 (+ 2370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:34.618813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:34.620624 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:34.620638 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:34.621695 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:34.621701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:34.621911 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:34.622734 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:10:34.622759 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:10:34.625650 (+ 2891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:10:34.628951 (+ 3301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:10:34.628955 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:10:34.628958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:10:34.629473 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:10:34.629908 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:10:34.629913 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:10:34.629917 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:10:34.630129 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:10:34.630969 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:10:34.630977 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:10:34.631304 (+ 327us) client_negotiation.cc:769] Sending connection context
0926 03:10:34.631549 (+ 245us) client_negotiation.cc:240] Negotiation successful
0926 03:10:34.632147 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1025,"mutex_wait_us":351,"spinlock_wait_cycles":15360,"thread_start_us":889,"threads_started":1}
I20260926 03:10:34.633219 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:34.610228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35919 (local address 127.0.97.1:60087)
0926 03:10:34.610910 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:34.610931 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:34.611065 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:34.614149 (+ 3084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:34.614161 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:34.615274 (+ 1113us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:10:34.616406 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:34.616437 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:34.618810 (+ 2373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:34.618815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:34.620629 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:34.620643 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:34.621640 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:34.621647 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:34.621918 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:34.622796 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:10:34.622813 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:10:34.625420 (+ 2607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:10:34.628919 (+ 3499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:10:34.628929 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:10:34.628946 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:10:34.629465 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:10:34.629919 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:10:34.629923 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:10:34.629925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:10:34.630139 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:10:34.630987 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:10:34.630992 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:10:34.631303 (+ 311us) client_negotiation.cc:769] Sending connection context
0926 03:10:34.631549 (+ 246us) client_negotiation.cc:240] Negotiation successful
0926 03:10:34.632298 (+ 749us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":453,"spinlock_wait_cycles":10240,"thread_start_us":366,"threads_started":1}
W20260926 03:10:34.636126 17113 leader_election.cc:341] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42c7171b04bf4c4d9650ef6e041aa932 (127.0.97.3:43637): Not found: Tablet not found: a4ccedce23b249719e11bd5a7d7aa4a2
W20260926 03:10:34.636432 17113 leader_election.cc:341] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ff233c21ff6f4757b49bfe0ad0c65e33 (127.0.97.2:35919): Not found: Tablet not found: a4ccedce23b249719e11bd5a7d7aa4a2
I20260926 03:10:34.636546 17113 leader_election.cc:302] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67353804c3504e4287cd5aa5efe6e8ac; no voters: 42c7171b04bf4c4d9650ef6e041aa932, ff233c21ff6f4757b49bfe0ad0c65e33
I20260926 03:10:34.636904 17494 raft_consensus.cc:2726] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 03:10:34.745872 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:34.744963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.3:40705)
0926 03:10:34.745483 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:34.745610 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":251,"threads_started":1}
I20260926 03:10:34.825171 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:34.824733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.1:33789)
0926 03:10:34.824907 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:34.825025 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 03:10:35.288740 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:35.287916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.2:36815)
0926 03:10:35.288447 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:35.288544 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":272,"threads_started":1}
I20260926 03:10:35.747646 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:35.746881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.3:37557)
0926 03:10:35.747350 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:35.747438 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":221,"threads_started":1}
I20260926 03:10:35.827054 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:35.826164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.1:38637)
0926 03:10:35.826763 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:35.826853 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":301,"threads_started":1}
I20260926 03:10:36.290609 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:36.289769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.2:56611)
0926 03:10:36.290293 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:36.290383 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":304,"threads_started":1}
I20260926 03:10:36.748171 17504 raft_consensus.cc:482] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:10:36.748365 17504 raft_consensus.cc:504] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67353804c3504e4287cd5aa5efe6e8ac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44023 } } peers { permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } peers { permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 } }
I20260926 03:10:36.749006 17504 leader_election.cc:288] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42c7171b04bf4c4d9650ef6e041aa932 (127.0.97.3:43637), ff233c21ff6f4757b49bfe0ad0c65e33 (127.0.97.2:35919)
I20260926 03:10:36.749439 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:36.748617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.3:39497)
0926 03:10:36.749108 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:36.749215 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":226,"threads_started":1}
W20260926 03:10:36.749974 17113 leader_election.cc:341] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42c7171b04bf4c4d9650ef6e041aa932 (127.0.97.3:43637): Not found: Tablet not found: a4ccedce23b249719e11bd5a7d7aa4a2
W20260926 03:10:36.750212 17113 leader_election.cc:341] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ff233c21ff6f4757b49bfe0ad0c65e33 (127.0.97.2:35919): Not found: Tablet not found: a4ccedce23b249719e11bd5a7d7aa4a2
I20260926 03:10:36.750294 17113 leader_election.cc:302] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67353804c3504e4287cd5aa5efe6e8ac; no voters: 42c7171b04bf4c4d9650ef6e041aa932, ff233c21ff6f4757b49bfe0ad0c65e33
I20260926 03:10:36.750626 17504 raft_consensus.cc:2726] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 03:10:36.828858 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:36.827998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.1:41579)
0926 03:10:36.828495 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:36.828662 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":237,"threads_started":1}
I20260926 03:10:37.292272 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:37.291517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.2:40793)
0926 03:10:37.291969 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:37.292076 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":206,"threads_started":1}
I20260926 03:10:37.751196 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:37.750435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.3:35919)
0926 03:10:37.750920 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:37.751004 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":274,"threads_started":1}
I20260926 03:10:37.830760 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:37.829922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.1:40595)
0926 03:10:37.830402 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:37.830566 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":256,"threads_started":1}
I20260926 03:10:38.294032 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:38.293260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.2:39743)
0926 03:10:38.293776 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:38.293861 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":311,"threads_started":1}
I20260926 03:10:38.752854 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:38.752129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.3:59669)
0926 03:10:38.752601 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:38.752686 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":274,"threads_started":1}
I20260926 03:10:38.832379 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:38.831645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.1:56151)
0926 03:10:38.832100 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:38.832203 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":228,"threads_started":1}
I20260926 03:10:39.295689 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:39.294916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.2:53243)
0926 03:10:39.295409 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:39.295502 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":283,"threads_started":1}
I20260926 03:10:39.652130 17514 raft_consensus.cc:482] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:10:39.652339 17514 raft_consensus.cc:504] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67353804c3504e4287cd5aa5efe6e8ac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44023 } } peers { permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } peers { permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 } }
I20260926 03:10:39.652961 17514 leader_election.cc:288] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42c7171b04bf4c4d9650ef6e041aa932 (127.0.97.3:43637), ff233c21ff6f4757b49bfe0ad0c65e33 (127.0.97.2:35919)
W20260926 03:10:39.654024 17113 leader_election.cc:341] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42c7171b04bf4c4d9650ef6e041aa932 (127.0.97.3:43637): Not found: Tablet not found: a4ccedce23b249719e11bd5a7d7aa4a2
W20260926 03:10:39.654246 17113 leader_election.cc:341] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ff233c21ff6f4757b49bfe0ad0c65e33 (127.0.97.2:35919): Not found: Tablet not found: a4ccedce23b249719e11bd5a7d7aa4a2
I20260926 03:10:39.654372 17113 leader_election.cc:302] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67353804c3504e4287cd5aa5efe6e8ac; no voters: 42c7171b04bf4c4d9650ef6e041aa932, ff233c21ff6f4757b49bfe0ad0c65e33
I20260926 03:10:39.654663 17514 raft_consensus.cc:2726] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 03:10:39.754878 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:39.753989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.3:39511)
0926 03:10:39.754590 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:39.754688 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":334,"threads_started":1}
I20260926 03:10:39.834461 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:39.833404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.1:43553)
0926 03:10:39.834031 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:39.834160 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":274,"threads_started":1}
I20260926 03:10:40.297647 17517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:40.296782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.2:41573)
0926 03:10:40.297344 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:40.297434 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":254,"threads_started":1}
I20260926 03:10:40.756803 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:40.755911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.3:60747)
0926 03:10:40.756512 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:40.756605 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":277,"threads_started":1}
I20260926 03:10:40.836213 17519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:40.835511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.1:35827)
0926 03:10:40.835960 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:40.836047 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":222,"threads_started":1}
I20260926 03:10:41.299479 17520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:41.298716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.2:42987)
0926 03:10:41.299207 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:41.299305 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":228,"threads_started":1}
I20260926 03:10:41.759200 17521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:41.758328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.3:41405)
0926 03:10:41.758879 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:41.759006 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":302,"threads_started":1}
I20260926 03:10:41.837982 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:41.837102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.1:43183)
0926 03:10:41.837668 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:41.837776 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":261,"threads_started":1}
I20260926 03:10:42.152383 17523 raft_consensus.cc:482] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:10:42.152683 17523 raft_consensus.cc:504] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67353804c3504e4287cd5aa5efe6e8ac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44023 } } peers { permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } peers { permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 } }
I20260926 03:10:42.153247 17523 leader_election.cc:288] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42c7171b04bf4c4d9650ef6e041aa932 (127.0.97.3:43637), ff233c21ff6f4757b49bfe0ad0c65e33 (127.0.97.2:35919)
W20260926 03:10:42.154261 17113 leader_election.cc:341] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42c7171b04bf4c4d9650ef6e041aa932 (127.0.97.3:43637): Not found: Tablet not found: a4ccedce23b249719e11bd5a7d7aa4a2
W20260926 03:10:42.154520 17113 leader_election.cc:341] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ff233c21ff6f4757b49bfe0ad0c65e33 (127.0.97.2:35919): Not found: Tablet not found: a4ccedce23b249719e11bd5a7d7aa4a2
I20260926 03:10:42.154624 17113 leader_election.cc:302] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67353804c3504e4287cd5aa5efe6e8ac; no voters: 42c7171b04bf4c4d9650ef6e041aa932, ff233c21ff6f4757b49bfe0ad0c65e33
I20260926 03:10:42.154973 17523 raft_consensus.cc:2726] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 03:10:42.301908 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:42.300914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.2:49385)
0926 03:10:42.301621 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:42.301730 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":430,"threads_started":1}
I20260926 03:10:42.761312 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:42.760515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.3:48989)
0926 03:10:42.761057 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:42.761149 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":254,"threads_started":1}
I20260926 03:10:42.839974 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:42.839168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.1:44773)
0926 03:10:42.839674 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:42.839763 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":274,"threads_started":1}
I20260926 03:10:43.303743 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:43.303020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.2:52793)
0926 03:10:43.303438 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:43.303562 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":219,"threads_started":1}
I20260926 03:10:43.763154 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:43.762393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.3:52005)
0926 03:10:43.762885 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:43.762970 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":273,"threads_started":1}
I20260926 03:10:43.841894 17529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:43.841039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.1:56665)
0926 03:10:43.841588 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:43.841681 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":282,"threads_started":1}
I20260926 03:10:43.910868 17530 raft_consensus.cc:482] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:10:43.911082 17530 raft_consensus.cc:504] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67353804c3504e4287cd5aa5efe6e8ac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44023 } } peers { permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } peers { permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 } }
I20260926 03:10:43.911736 17530 leader_election.cc:288] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42c7171b04bf4c4d9650ef6e041aa932 (127.0.97.3:43637), ff233c21ff6f4757b49bfe0ad0c65e33 (127.0.97.2:35919)
W20260926 03:10:43.912796 17113 leader_election.cc:341] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42c7171b04bf4c4d9650ef6e041aa932 (127.0.97.3:43637): Not found: Tablet not found: a4ccedce23b249719e11bd5a7d7aa4a2
W20260926 03:10:43.912997 17113 leader_election.cc:341] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ff233c21ff6f4757b49bfe0ad0c65e33 (127.0.97.2:35919): Not found: Tablet not found: a4ccedce23b249719e11bd5a7d7aa4a2
I20260926 03:10:43.913098 17113 leader_election.cc:302] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67353804c3504e4287cd5aa5efe6e8ac; no voters: 42c7171b04bf4c4d9650ef6e041aa932, ff233c21ff6f4757b49bfe0ad0c65e33
I20260926 03:10:43.913409 17530 raft_consensus.cc:2726] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 03:10:44.305716 17531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:44.304887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.2:44965)
0926 03:10:44.305442 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:44.305537 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":259,"threads_started":1}
I20260926 03:10:44.765097 17532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:44.764238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.3:33275)
0926 03:10:44.764811 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:44.764929 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":296,"threads_started":1}
I20260926 03:10:44.843707 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:44.842911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.1:47851)
0926 03:10:44.843390 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:44.843532 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":235,"threads_started":1}
I20260926 03:10:45.307801 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:45.306923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.2:53743)
0926 03:10:45.307438 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:45.307597 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":275,"threads_started":1}
I20260926 03:10:45.766969 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:45.766100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.3:51673)
0926 03:10:45.766669 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:45.766761 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":272,"threads_started":1}
I20260926 03:10:45.845546 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:45.844729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.1:32809)
0926 03:10:45.845262 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:45.845354 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":274,"threads_started":1}
I20260926 03:10:46.309616 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:46.308889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.2:38865)
0926 03:10:46.309344 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:46.309434 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":223,"threads_started":1}
I20260926 03:10:46.768775 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:46.767973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.3:60405)
0926 03:10:46.768504 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:46.768599 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":267,"threads_started":1}
I20260926 03:10:46.847296 17539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:46.846487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.1:42341)
0926 03:10:46.847012 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:46.847099 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":245,"threads_started":1}
I20260926 03:10:47.311991 17540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:47.310682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.2:36923)
0926 03:10:47.311253 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:47.311487 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":305,"threads_started":1}
I20260926 03:10:47.770781 17541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:47.769861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.3:44437)
0926 03:10:47.770431 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:47.770568 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":262,"threads_started":1}
I20260926 03:10:47.849502 17542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:47.848690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.1:43355)
0926 03:10:47.849177 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:47.849271 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":243,"threads_started":1}
I20260926 03:10:48.314246 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:48.313493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.2:56383)
0926 03:10:48.313991 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:48.314085 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":242,"threads_started":1}
I20260926 03:10:48.710711 17544 raft_consensus.cc:482] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:10:48.710911 17544 raft_consensus.cc:504] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67353804c3504e4287cd5aa5efe6e8ac" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44023 } } peers { permanent_uuid: "42c7171b04bf4c4d9650ef6e041aa932" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } peers { permanent_uuid: "ff233c21ff6f4757b49bfe0ad0c65e33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35919 } }
I20260926 03:10:48.711544 17544 leader_election.cc:288] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42c7171b04bf4c4d9650ef6e041aa932 (127.0.97.3:43637), ff233c21ff6f4757b49bfe0ad0c65e33 (127.0.97.2:35919)
W20260926 03:10:48.712607 17113 leader_election.cc:341] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42c7171b04bf4c4d9650ef6e041aa932 (127.0.97.3:43637): Not found: Tablet not found: a4ccedce23b249719e11bd5a7d7aa4a2
W20260926 03:10:48.712822 17113 leader_election.cc:341] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ff233c21ff6f4757b49bfe0ad0c65e33 (127.0.97.2:35919): Not found: Tablet not found: a4ccedce23b249719e11bd5a7d7aa4a2
I20260926 03:10:48.712919 17113 leader_election.cc:302] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67353804c3504e4287cd5aa5efe6e8ac; no voters: 42c7171b04bf4c4d9650ef6e041aa932, ff233c21ff6f4757b49bfe0ad0c65e33
I20260926 03:10:48.713214 17544 raft_consensus.cc:2726] T a4ccedce23b249719e11bd5a7d7aa4a2 P 67353804c3504e4287cd5aa5efe6e8ac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 03:10:48.743738 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 17055
I20260926 03:10:48.765972 17545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:48.744835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33336 (local address 127.0.97.1:44023)
0926 03:10:48.745609 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:48.745620 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:48.745812 (+ 192us) server_negotiation.cc:407] Connection header received
0926 03:10:48.746040 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:48.746050 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:48.746298 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:48.746449 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:48.747727 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:48.748736 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:48.750074 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:48.750684 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:48.750784 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:48.759551 (+ 8767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:48.759568 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:48.759583 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:10:48.759627 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:48.762792 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:48.763597 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:48.763606 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:48.763608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:48.763779 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:48.764227 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:48.764231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:48.764234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:48.764730 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:48.764986 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:48.765305 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 03:10:48.765590 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":299,"threads_started":1}
I20260926 03:10:48.767349 17220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:10:48.772801 17547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:48.772042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.3:47647)
0926 03:10:48.772528 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:48.772628 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":235,"threads_started":1}
I20260926 03:10:48.859422 17548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:48.858316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39617 (local address 127.0.97.1:53489)
0926 03:10:48.858951 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:48.859109 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":319,"threads_started":1}
I20260926 03:10:48.863010 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 17055
I20260926 03:10:48.878422 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 17229
I20260926 03:10:48.892081 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:48.879222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45874 (local address 127.0.97.2:35919)
0926 03:10:48.879754 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:48.879759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:48.879774 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:10:48.879830 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:48.879833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:48.879899 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:48.879998 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:48.881276 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:48.882161 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:48.883525 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:48.883981 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:48.884095 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:48.886697 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:48.886703 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:48.886706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:10:48.886745 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:48.889343 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:48.890084 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:48.890087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:48.890090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:48.890219 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:48.890759 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:48.890762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:48.890764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:48.891047 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:48.891259 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:48.891562 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 03:10:48.891739 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":339,"threads_started":1}
I20260926 03:10:48.892765 17352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:10:48.979941 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 17229
I20260926 03:10:48.994372 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 17360
I20260926 03:10:49.008762 17551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:48.995214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42908 (local address 127.0.97.3:43637)
0926 03:10:48.995735 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:48.995740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:48.995786 (+ 46us) server_negotiation.cc:407] Connection header received
0926 03:10:48.995845 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:48.995849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:48.995941 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:48.996067 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:48.997516 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:48.998316 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:48.999671 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:49.000113 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:49.000222 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:49.002817 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:49.002830 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:49.002835 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:10:49.002880 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:49.005684 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:49.006488 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:49.006494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:49.006614 (+ 120us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:49.006862 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:49.007258 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:49.007263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:49.007265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:49.007608 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:49.007837 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:49.008119 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 03:10:49.008313 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":256,"threads_started":1}
I20260926 03:10:49.009469 17483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:10:49.097458 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 17360
2026-09-26T03:10:49Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64322 ms)
[----------] 30 tests from SecurityITest (298993 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:10:49 dist-test-slave-hmbw krb5kdc[17555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:10:49 dist-test-slave-hmbw krb5kdc[17555](info): set up 2 sockets
Sep 26 03:10:49 dist-test-slave-hmbw krb5kdc[17555](info): commencing operation
krb5kdc: starting...
W20260926 03:10:50.126428 388 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:10:50 dist-test-slave-hmbw krb5kdc[17555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:10:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:10:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:10:50.378249 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36785
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:10:50.743423 17568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:10:50.743779 17568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:10:50.743887 17568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:10:50.751778 17568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:10:50.751906 17568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:10:50.751952 17568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:10:50.752012 17568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:10:50.752056 17568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:10:50.762953 17568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:10:50.764310 17568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:10:50.765692 17568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:10:50.778757 17573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:50.778884 17574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:50.778808 17576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:10:50.779122 17568 server_base.cc:1034] running on GCE node
I20260926 03:10:50.780153 17568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:10:50.781744 17568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:10:50.782977 17568 hybrid_clock.cc:648] HybridClock initialized: now 1790392250782899 us; error 66 us; skew 500 ppm
I20260926 03:10:50.783351 17568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:10:50 dist-test-slave-hmbw krb5kdc[17555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:10:50.786752 17568 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:10:50.788400 17568 webserver.cc:460] Webserver started at http://127.0.97.62:33019/ using document root <none> and password file <none>
I20260926 03:10:50.789108 17568 fs_manager.cc:359] Metadata directory not provided
I20260926 03:10:50.789227 17568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:10:50.789531 17568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:10:50.792699 17568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "be132b38c1f24693a3465b075c43a8c9"
format_stamp: "Formatted at 2026-09-26 03:10:50 on dist-test-slave-hmbw"
server_key: "a969f6cdc1f813fa763f7fef7e05fb81"
server_key_iv: "5e212563d71523b072c4660413496a1e"
server_key_version: "encryptionkey@0"
I20260926 03:10:50.793538 17568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "be132b38c1f24693a3465b075c43a8c9"
format_stamp: "Formatted at 2026-09-26 03:10:50 on dist-test-slave-hmbw"
server_key: "a969f6cdc1f813fa763f7fef7e05fb81"
server_key_iv: "5e212563d71523b072c4660413496a1e"
server_key_version: "encryptionkey@0"
I20260926 03:10:50.799208 17568 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:10:50.803130 17583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:50.804666 17568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 03:10:50.804854 17568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "be132b38c1f24693a3465b075c43a8c9"
format_stamp: "Formatted at 2026-09-26 03:10:50 on dist-test-slave-hmbw"
server_key: "a969f6cdc1f813fa763f7fef7e05fb81"
server_key_iv: "5e212563d71523b072c4660413496a1e"
server_key_version: "encryptionkey@0"
I20260926 03:10:50.805064 17568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:10:50.821465 17568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:10:50.825968 17568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:10:50.826261 17568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:10:50.844843 17568 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36785
I20260926 03:10:50.844920 17634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36785 every 8 connection(s)
I20260926 03:10:50.846386 17568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:10:50.850581 17635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:10:50.853391 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 17568
I20260926 03:10:50.853602 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:10:50.854269 388 external_mini_cluster.cc:1428] Setting key 8343dce7ebd239d05c1555c5542fd1ab
I20260926 03:10:50.864344 17635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be132b38c1f24693a3465b075c43a8c9: Bootstrap starting.
I20260926 03:10:50.868355 17635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be132b38c1f24693a3465b075c43a8c9: Neither blocks nor log segments found. Creating new log.
I20260926 03:10:50.869436 17635 log.cc:824] T 00000000000000000000000000000000 P be132b38c1f24693a3465b075c43a8c9: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:10:50 dist-test-slave-hmbw krb5kdc[17555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:10:50.872644 17635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be132b38c1f24693a3465b075c43a8c9: No bootstrap required, opened a new log
I20260926 03:10:50.879809 17638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:50.856603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45292 (local address 127.0.97.62:36785)
0926 03:10:50.857450 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:50.857466 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:50.857520 (+ 54us) server_negotiation.cc:407] Connection header received
0926 03:10:50.859854 (+ 2334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:50.859889 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:50.860496 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:50.861014 (+ 518us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:50.862560 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:50.864541 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:50.866549 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:50.867619 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:50.867844 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:50.872033 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:50.872042 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:50.872058 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:10:50.872097 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:50.875296 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:50.876078 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:50.876090 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:50.876098 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:50.876297 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:50.876697 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:50.876704 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:50.876706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:50.877382 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:50.877700 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:50.878181 (+ 481us) server_negotiation.cc:299] Negotiation successful
0926 03:10:50.878685 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":368,"threads_started":1}
I20260926 03:10:50.880918 17635 raft_consensus.cc:348] T 00000000000000000000000000000000 P be132b38c1f24693a3465b075c43a8c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be132b38c1f24693a3465b075c43a8c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36785 } }
I20260926 03:10:50.881446 17635 raft_consensus.cc:374] T 00000000000000000000000000000000 P be132b38c1f24693a3465b075c43a8c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:10:50.881565 17635 raft_consensus.cc:729] T 00000000000000000000000000000000 P be132b38c1f24693a3465b075c43a8c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be132b38c1f24693a3465b075c43a8c9, State: Initialized, Role: FOLLOWER
I20260926 03:10:50.882284 17635 consensus_queue.cc:260] T 00000000000000000000000000000000 P be132b38c1f24693a3465b075c43a8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be132b38c1f24693a3465b075c43a8c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36785 } }
I20260926 03:10:50.882722 17635 raft_consensus.cc:388] T 00000000000000000000000000000000 P be132b38c1f24693a3465b075c43a8c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:10:50.882941 17635 raft_consensus.cc:482] T 00000000000000000000000000000000 P be132b38c1f24693a3465b075c43a8c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:10:50.883131 17635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be132b38c1f24693a3465b075c43a8c9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:10:50.885581 17635 raft_consensus.cc:504] T 00000000000000000000000000000000 P be132b38c1f24693a3465b075c43a8c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be132b38c1f24693a3465b075c43a8c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36785 } }
I20260926 03:10:50.886123 17635 leader_election.cc:302] T 00000000000000000000000000000000 P be132b38c1f24693a3465b075c43a8c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be132b38c1f24693a3465b075c43a8c9; no voters:
I20260926 03:10:50.886785 17635 leader_election.cc:288] T 00000000000000000000000000000000 P be132b38c1f24693a3465b075c43a8c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:10:50.886916 17640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be132b38c1f24693a3465b075c43a8c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:10:50.887576 17640 raft_consensus.cc:686] T 00000000000000000000000000000000 P be132b38c1f24693a3465b075c43a8c9 [term 1 LEADER]: Becoming Leader. State: Replica: be132b38c1f24693a3465b075c43a8c9, State: Running, Role: LEADER
I20260926 03:10:50.888134 17640 consensus_queue.cc:237] T 00000000000000000000000000000000 P be132b38c1f24693a3465b075c43a8c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be132b38c1f24693a3465b075c43a8c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36785 } }
I20260926 03:10:50.888875 17635 sys_catalog.cc:564] T 00000000000000000000000000000000 P be132b38c1f24693a3465b075c43a8c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:10:50.891402 17642 sys_catalog.cc:455] T 00000000000000000000000000000000 P be132b38c1f24693a3465b075c43a8c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be132b38c1f24693a3465b075c43a8c9. Latest consensus state: current_term: 1 leader_uuid: "be132b38c1f24693a3465b075c43a8c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be132b38c1f24693a3465b075c43a8c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36785 } } }
I20260926 03:10:50.891729 17642 sys_catalog.cc:458] T 00000000000000000000000000000000 P be132b38c1f24693a3465b075c43a8c9 [sys.catalog]: This master's current role is: LEADER
I20260926 03:10:50.892597 17648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:10:50.892386 17641 sys_catalog.cc:455] T 00000000000000000000000000000000 P be132b38c1f24693a3465b075c43a8c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be132b38c1f24693a3465b075c43a8c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be132b38c1f24693a3465b075c43a8c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36785 } } }
I20260926 03:10:50.892809 17641 sys_catalog.cc:458] T 00000000000000000000000000000000 P be132b38c1f24693a3465b075c43a8c9 [sys.catalog]: This master's current role is: LEADER
I20260926 03:10:50.899190 17648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:10:50.908984 17648 catalog_manager.cc:1347] Generated new cluster ID: 41d4402e34d3432f81287f5dcf7c4cca
I20260926 03:10:50.909126 17648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:10:50.918901 17648 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:10:50.920807 17648 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:10:50.931730 17648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be132b38c1f24693a3465b075c43a8c9: Generated new TSK 0
I20260926 03:10:50.932945 17648 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:10:50.991165 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36785
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:10:51.341327 17661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:10:51.341683 17661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:10:51.341832 17661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:10:51.349861 17661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:10:51.350024 17661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:10:51.350188 17661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:10:51.360965 17661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:10:51.362241 17661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:10:51.363636 17661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:10:51.377477 17666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:51.377420 17667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:51.377455 17669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:10:51.378134 17661 server_base.cc:1034] running on GCE node
I20260926 03:10:51.378937 17661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:10:51.379979 17661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:10:51.381210 17661 hybrid_clock.cc:648] HybridClock initialized: now 1790392251381149 us; error 46 us; skew 500 ppm
I20260926 03:10:51.381539 17661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:10:51 dist-test-slave-hmbw krb5kdc[17555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:10:51.384680 17661 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:10:51.386253 17661 webserver.cc:460] Webserver started at http://127.0.97.1:33689/ using document root <none> and password file <none>
I20260926 03:10:51.386997 17661 fs_manager.cc:359] Metadata directory not provided
I20260926 03:10:51.387104 17661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:10:51.387435 17661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:10:51.390690 17661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-0/data/instance:
uuid: "3ba3935f5f3040d9843dce0bd44a8fe3"
format_stamp: "Formatted at 2026-09-26 03:10:51 on dist-test-slave-hmbw"
server_key: "63c37af75f8896d648b3a0432cb33347"
server_key_iv: "8775f09ce6d737941afbd21ead17b3e7"
server_key_version: "encryptionkey@0"
I20260926 03:10:51.391513 17661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3ba3935f5f3040d9843dce0bd44a8fe3"
format_stamp: "Formatted at 2026-09-26 03:10:51 on dist-test-slave-hmbw"
server_key: "63c37af75f8896d648b3a0432cb33347"
server_key_iv: "8775f09ce6d737941afbd21ead17b3e7"
server_key_version: "encryptionkey@0"
I20260926 03:10:51.397055 17661 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:10:51.400857 17676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:51.402484 17661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:10:51.402746 17661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "3ba3935f5f3040d9843dce0bd44a8fe3"
format_stamp: "Formatted at 2026-09-26 03:10:51 on dist-test-slave-hmbw"
server_key: "63c37af75f8896d648b3a0432cb33347"
server_key_iv: "8775f09ce6d737941afbd21ead17b3e7"
server_key_version: "encryptionkey@0"
I20260926 03:10:51.402956 17661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:10:51.436113 17661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:10:51.440608 17661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:10:51.440887 17661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:10:51.441913 17661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:10:51.443459 17661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:10:51.443537 17661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:51.443672 17661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:10:51.443722 17661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:51.476889 17661 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46115
I20260926 03:10:51.476959 17788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46115 every 8 connection(s)
I20260926 03:10:51.478161 17661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:10:51.479470 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 17661
I20260926 03:10:51.479672 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:10:51.480301 388 external_mini_cluster.cc:1428] Setting key 49e950dd75a2bcfc62998a690699196d
Sep 26 03:10:51 dist-test-slave-hmbw krb5kdc[17555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:10:51.500120 17791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:51.480979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54527 (local address 127.0.97.62:36785)
0926 03:10:51.481726 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:51.481732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:51.482699 (+ 967us) server_negotiation.cc:407] Connection header received
0926 03:10:51.484757 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:51.484763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:51.484862 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:51.484959 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:51.486651 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:51.487427 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:51.489188 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:51.489825 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:51.489993 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:51.493857 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:51.493863 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:51.493867 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:10:51.493905 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:51.496525 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:51.497518 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:51.497523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:51.497526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:51.497651 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:51.498163 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:51.498167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:51.498169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:51.498463 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:51.498627 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:51.499576 (+ 949us) server_negotiation.cc:299] Negotiation successful
0926 03:10:51.499837 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":494,"threads_started":1}
I20260926 03:10:51.501607 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:51.481571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36785 (local address 127.0.97.1:54527)
0926 03:10:51.482477 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:51.482583 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:51.484421 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:51.485249 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:51.485258 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:51.485822 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:10:51.486448 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:51.486467 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:51.487613 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:51.487618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:51.489028 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:51.489044 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:51.490110 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:51.490116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:51.490401 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:51.491085 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:10:51.491109 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:10:51.493537 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:10:51.496705 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:10:51.496715 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:10:51.496737 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:10:51.497363 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:10:51.497756 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:10:51.497760 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:10:51.497764 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:10:51.498021 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:10:51.498615 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:10:51.498624 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:10:51.499099 (+ 475us) client_negotiation.cc:769] Sending connection context
0926 03:10:51.499424 (+ 325us) client_negotiation.cc:240] Negotiation successful
0926 03:10:51.500048 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":334,"threads_started":1}
I20260926 03:10:51.503647 17789 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36785
I20260926 03:10:51.503999 17789 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:10:51.504854 17789 heartbeater.cc:511] Master 127.0.97.62:36785 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 03:10:51.507525 17600 ts_manager.cc:194] Registered new tserver with Master: 3ba3935f5f3040d9843dce0bd44a8fe3 (127.0.97.1:46115)
I20260926 03:10:51.509684 17600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:10:51.538286 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36785
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:10:51.880733 17795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:10:51.881040 17795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:10:51.881188 17795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:10:51.889259 17795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:10:51.889411 17795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:10:51.889566 17795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:10:51.899911 17795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:10:51.901307 17795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:10:51.902690 17795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:10:51.916460 17801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:51.916460 17800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:51.917016 17803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:10:51.917448 17795 server_base.cc:1034] running on GCE node
I20260926 03:10:51.918112 17795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:10:51.919193 17795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:10:51.920444 17795 hybrid_clock.cc:648] HybridClock initialized: now 1790392251920369 us; error 60 us; skew 500 ppm
I20260926 03:10:51.920797 17795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:10:51 dist-test-slave-hmbw krb5kdc[17555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:10:51.924289 17795 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:10:51.926038 17795 webserver.cc:460] Webserver started at http://127.0.97.2:46227/ using document root <none> and password file <none>
I20260926 03:10:51.926874 17795 fs_manager.cc:359] Metadata directory not provided
I20260926 03:10:51.926999 17795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:10:51.927317 17795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:10:51.930430 17795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-1/data/instance:
uuid: "e42525cf268a4c328510560aa9934bd9"
format_stamp: "Formatted at 2026-09-26 03:10:51 on dist-test-slave-hmbw"
server_key: "70d74c51085c79429be76ad26406e304"
server_key_iv: "b6c577c29a48007dd3c460d86570c60f"
server_key_version: "encryptionkey@0"
I20260926 03:10:51.931371 17795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e42525cf268a4c328510560aa9934bd9"
format_stamp: "Formatted at 2026-09-26 03:10:51 on dist-test-slave-hmbw"
server_key: "70d74c51085c79429be76ad26406e304"
server_key_iv: "b6c577c29a48007dd3c460d86570c60f"
server_key_version: "encryptionkey@0"
I20260926 03:10:51.937067 17795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 03:10:51.940934 17810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:51.942490 17795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:10:51.942716 17795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "e42525cf268a4c328510560aa9934bd9"
format_stamp: "Formatted at 2026-09-26 03:10:51 on dist-test-slave-hmbw"
server_key: "70d74c51085c79429be76ad26406e304"
server_key_iv: "b6c577c29a48007dd3c460d86570c60f"
server_key_version: "encryptionkey@0"
I20260926 03:10:51.942932 17795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:10:51.959656 17795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:10:51.964323 17795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:10:51.964632 17795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:10:51.965664 17795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:10:51.967183 17795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:10:51.967263 17795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:51.967383 17795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:10:51.967434 17795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:52.005484 17795 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36803
I20260926 03:10:52.005527 17922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36803 every 8 connection(s)
I20260926 03:10:52.006958 17795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:10:52.013824 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 17795
I20260926 03:10:52.013998 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:10:52.014580 388 external_mini_cluster.cc:1428] Setting key 5afd667b22765368b1cd40f84e2cc92e
Sep 26 03:10:52 dist-test-slave-hmbw krb5kdc[17555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:10:52.031891 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:52.010046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36081 (local address 127.0.97.62:36785)
0926 03:10:52.010556 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:52.010562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:52.011799 (+ 1237us) server_negotiation.cc:407] Connection header received
0926 03:10:52.013775 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:52.013781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:52.013881 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:52.013980 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:52.015944 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.016765 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.019493 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.019977 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.020109 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:52.024734 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:52.024742 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:52.024746 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:10:52.024796 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:52.027854 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:52.029052 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:52.029058 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:52.029060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:52.029235 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:52.029828 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:52.029833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:52.029836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:52.030162 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:52.030287 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:52.031311 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0926 03:10:52.031604 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":214,"threads_started":1}
I20260926 03:10:52.033157 17926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:52.010722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36785 (local address 127.0.97.2:36081)
0926 03:10:52.011615 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:52.011660 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:52.013422 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:52.014262 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:52.014271 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:52.014969 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:10:52.015746 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:52.015768 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.016969 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.016975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:52.019179 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:52.019202 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.020245 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.020252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:52.020603 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:52.021414 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:10:52.021448 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:10:52.024364 (+ 2916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:10:52.028113 (+ 3749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:10:52.028125 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:10:52.028145 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:10:52.028842 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:10:52.029359 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:10:52.029364 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:10:52.029368 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:10:52.029656 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:10:52.030293 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:10:52.030302 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:10:52.030812 (+ 510us) client_negotiation.cc:769] Sending connection context
0926 03:10:52.031136 (+ 324us) client_negotiation.cc:240] Negotiation successful
0926 03:10:52.031704 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":317,"threads_started":1}
I20260926 03:10:52.034974 17923 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36785
I20260926 03:10:52.035326 17923 heartbeater.cc:464] Registering TS with master...
I20260926 03:10:52.036167 17923 heartbeater.cc:511] Master 127.0.97.62:36785 requested a full tablet report, sending...
I20260926 03:10:52.038087 17600 ts_manager.cc:194] Registered new tserver with Master: e42525cf268a4c328510560aa9934bd9 (127.0.97.2:36803)
I20260926 03:10:52.039121 17600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36081
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:10:52.074996 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36785
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:10:52.431787 17929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:10:52.432108 17929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:10:52.432216 17929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:10:52.439792 17929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:10:52.439922 17929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:10:52.440063 17929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:10:52.450812 17929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:10:52.452158 17929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:10:52.453557 17929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:10:52.467331 17935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:52.467554 17934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:52.467289 17937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:10:52.468175 17929 server_base.cc:1034] running on GCE node
I20260926 03:10:52.468876 17929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:10:52.469951 17929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:10:52.471200 17929 hybrid_clock.cc:648] HybridClock initialized: now 1790392252471130 us; error 64 us; skew 500 ppm
I20260926 03:10:52.471639 17929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:10:52 dist-test-slave-hmbw krb5kdc[17555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:10:52.475220 17929 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:10:52.477134 17929 webserver.cc:460] Webserver started at http://127.0.97.3:33247/ using document root <none> and password file <none>
I20260926 03:10:52.477952 17929 fs_manager.cc:359] Metadata directory not provided
I20260926 03:10:52.478087 17929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:10:52.478417 17929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:10:52.481668 17929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-2/data/instance:
uuid: "6ae856f1f1fc48608eed00e939a33235"
format_stamp: "Formatted at 2026-09-26 03:10:52 on dist-test-slave-hmbw"
server_key: "d42fe3867130662ed3ff615479a384a1"
server_key_iv: "32950c20134138b9c07653abbdff71b8"
server_key_version: "encryptionkey@0"
I20260926 03:10:52.482594 17929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6ae856f1f1fc48608eed00e939a33235"
format_stamp: "Formatted at 2026-09-26 03:10:52 on dist-test-slave-hmbw"
server_key: "d42fe3867130662ed3ff615479a384a1"
server_key_iv: "32950c20134138b9c07653abbdff71b8"
server_key_version: "encryptionkey@0"
I20260926 03:10:52.488297 17929 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 03:10:52.492537 17944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:52.494272 17929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:10:52.494542 17929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "6ae856f1f1fc48608eed00e939a33235"
format_stamp: "Formatted at 2026-09-26 03:10:52 on dist-test-slave-hmbw"
server_key: "d42fe3867130662ed3ff615479a384a1"
server_key_iv: "32950c20134138b9c07653abbdff71b8"
server_key_version: "encryptionkey@0"
I20260926 03:10:52.494799 17929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:10:52.513922 17789 heartbeater.cc:503] Master 127.0.97.62:36785 was elected leader, sending a full tablet report...
I20260926 03:10:52.531589 17929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:10:52.536396 17929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:10:52.536677 17929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:10:52.537644 17929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:10:52.539101 17929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:10:52.539183 17929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:52.539310 17929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:10:52.539364 17929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:52.573539 17929 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34767
I20260926 03:10:52.573592 18056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34767 every 8 connection(s)
I20260926 03:10:52.575016 17929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:10:52.581032 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 17929
I20260926 03:10:52.581215 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:10:52.581835 388 external_mini_cluster.cc:1428] Setting key fe05c9ac5b1a4c04f9d54b7e5389ae8b
Sep 26 03:10:52 dist-test-slave-hmbw krb5kdc[17555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:10:52.596860 18059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:52.577758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52877 (local address 127.0.97.62:36785)
0926 03:10:52.578242 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:52.578247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:52.579294 (+ 1047us) server_negotiation.cc:407] Connection header received
0926 03:10:52.581591 (+ 2297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:52.581597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:52.581680 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:52.581815 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:52.583841 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.584743 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.586457 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.586941 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.587052 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:52.590836 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:52.590841 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:52.590845 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:10:52.590881 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:52.593439 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:52.594336 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:52.594340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:52.594342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:52.594475 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:52.594950 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:52.594954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:52.594956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:52.595241 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:52.595383 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:52.596265 (+ 882us) server_negotiation.cc:299] Negotiation successful
0926 03:10:52.596488 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":258,"threads_started":1}
I20260926 03:10:52.597906 18060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:52.578346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36785 (local address 127.0.97.3:52877)
0926 03:10:52.579123 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:52.579162 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:52.581160 (+ 1998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:52.582123 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:52.582137 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:52.582850 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:10:52.583626 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:52.583656 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.584910 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.584915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:52.586307 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:52.586322 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.587180 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.587187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:52.587419 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:52.588118 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:10:52.588147 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:10:52.590600 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:10:52.593614 (+ 3014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:10:52.593623 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:10:52.593638 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:10:52.594208 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:10:52.594621 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:10:52.594624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:10:52.594627 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:10:52.594815 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:10:52.595359 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:10:52.595368 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:10:52.595827 (+ 459us) client_negotiation.cc:769] Sending connection context
0926 03:10:52.596130 (+ 303us) client_negotiation.cc:240] Negotiation successful
0926 03:10:52.596661 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":333,"threads_started":1}
I20260926 03:10:52.599264 18057 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36785
I20260926 03:10:52.599615 18057 heartbeater.cc:464] Registering TS with master...
I20260926 03:10:52.600347 18057 heartbeater.cc:511] Master 127.0.97.62:36785 requested a full tablet report, sending...
I20260926 03:10:52.601789 17600 ts_manager.cc:194] Registered new tserver with Master: 6ae856f1f1fc48608eed00e939a33235 (127.0.97.3:34767)
I20260926 03:10:52.602705 17600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52877
I20260926 03:10:52.610936 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:10:52.629076 18059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:52.615679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45296 (local address 127.0.97.62:36785)
0926 03:10:52.615938 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:52.615945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:52.616192 (+ 247us) server_negotiation.cc:407] Connection header received
0926 03:10:52.616373 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:52.616379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:52.616468 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:52.616562 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:52.618126 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.618955 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.620301 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.620716 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.620841 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:52.623372 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:52.623378 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:52.623381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:10:52.623420 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:52.626227 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:52.627063 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:52.627068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:52.627071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:52.627215 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:52.627602 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:52.627606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:52.627608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:52.627947 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:52.628193 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:52.628515 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 03:10:52.628753 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260926 03:10:52.633700 17600 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:10:52.640862 17600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:10:52.676342 18070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:52.654064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46115 (local address 127.0.0.1:59412)
0926 03:10:52.666352 (+ 12288us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:52.666403 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:52.666625 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:52.668275 (+ 1650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:52.668283 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:52.668360 (+ 77us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:10:52.668948 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:52.668964 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.671222 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.671228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:52.673259 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:52.673275 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.675207 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.675214 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:52.675437 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:52.675504 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 03:10:52.675619 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 03:10:52.675845 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12006,"mutex_wait_us":34,"spinlock_wait_cycles":26240,"thread_start_us":11893,"threads_started":1}
I20260926 03:10:52.676736 18071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:52.654190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59412 (local address 127.0.97.1:46115)
0926 03:10:52.667744 (+ 13554us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:52.667751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:52.667775 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:10:52.667858 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:52.667864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:52.668053 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:52.668944 (+ 891us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:10:52.669203 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.671059 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.673451 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.674983 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.675935 (+ 952us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:52.676160 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:52.676238 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 03:10:52.676423 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13367,"thread_start_us":13219,"threads_started":1}
I20260926 03:10:52.676960 18073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:52.654821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36918 (local address 127.0.97.3:34767)
0926 03:10:52.665927 (+ 11106us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:52.665938 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:52.667538 (+ 1600us) server_negotiation.cc:407] Connection header received
0926 03:10:52.667668 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:52.667677 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:52.667967 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:52.668183 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:10:52.669661 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.671308 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.673977 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.675438 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.675560 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:52.675794 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:52.676451 (+ 657us) server_negotiation.cc:299] Negotiation successful
0926 03:10:52.676618 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10905,"thread_start_us":10734,"threads_started":1}
I20260926 03:10:52.677172 18074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:52.654274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34767 (local address 127.0.0.1:36918)
0926 03:10:52.667248 (+ 12974us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:52.667265 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:52.667401 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:52.668275 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:52.668283 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:52.668364 (+ 81us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:10:52.669427 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:52.669440 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.671445 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.671451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:52.673777 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:52.673795 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.675677 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.675684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:52.675874 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:52.675918 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 03:10:52.676013 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 03:10:52.676153 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12838,"mutex_wait_us":75,"spinlock_wait_cycles":37888,"thread_start_us":14337,"threads_started":1}
I20260926 03:10:52.678035 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:52.656723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36803 (local address 127.0.0.1:55512)
0926 03:10:52.668936 (+ 12213us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:52.668955 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:52.669198 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:52.670121 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:52.670126 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:52.670152 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:10:52.670425 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:52.670432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.672030 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.672036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:52.674217 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:52.674232 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.676405 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.676413 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:52.677424 (+ 1011us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:52.677458 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 03:10:52.677543 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 03:10:52.677679 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12062,"mutex_wait_us":47,"thread_start_us":10474,"threads_started":1}
I20260926 03:10:52.678109 18075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:52.657326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55512 (local address 127.0.97.2:36803)
0926 03:10:52.667225 (+ 9899us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:52.667235 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:52.669076 (+ 1841us) server_negotiation.cc:407] Connection header received
0926 03:10:52.669414 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:52.669423 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:52.669661 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:52.669844 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:10:52.670597 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.671869 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.674451 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.675916 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.676041 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:52.676241 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:52.677627 (+ 1386us) server_negotiation.cc:299] Negotiation successful
0926 03:10:52.677801 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9742,"thread_start_us":9596,"threads_started":1}
I20260926 03:10:52.681186 17858 tablet_service.cc:1467] Processing CreateTablet for tablet ad38425ea13a417ba2ede1e5582702bf (DEFAULT_TABLE table=test-table [id=3151a8e680124791bfa12ef16449aa91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:10:52.681010 17724 tablet_service.cc:1467] Processing CreateTablet for tablet ad38425ea13a417ba2ede1e5582702bf (DEFAULT_TABLE table=test-table [id=3151a8e680124791bfa12ef16449aa91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:10:52.681010 17992 tablet_service.cc:1467] Processing CreateTablet for tablet ad38425ea13a417ba2ede1e5582702bf (DEFAULT_TABLE table=test-table [id=3151a8e680124791bfa12ef16449aa91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:10:52.682446 17858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad38425ea13a417ba2ede1e5582702bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:10:52.682580 17724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad38425ea13a417ba2ede1e5582702bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:10:52.682725 17992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad38425ea13a417ba2ede1e5582702bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:10:52.694629 18076 tablet_bootstrap.cc:492] T ad38425ea13a417ba2ede1e5582702bf P e42525cf268a4c328510560aa9934bd9: Bootstrap starting.
I20260926 03:10:52.697396 18077 tablet_bootstrap.cc:492] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3: Bootstrap starting.
I20260926 03:10:52.700042 18076 tablet_bootstrap.cc:654] T ad38425ea13a417ba2ede1e5582702bf P e42525cf268a4c328510560aa9934bd9: Neither blocks nor log segments found. Creating new log.
I20260926 03:10:52.700968 18077 tablet_bootstrap.cc:654] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3: Neither blocks nor log segments found. Creating new log.
I20260926 03:10:52.701378 18076 log.cc:824] T ad38425ea13a417ba2ede1e5582702bf P e42525cf268a4c328510560aa9934bd9: Log is configured to *not* fsync() on all Append() calls
I20260926 03:10:52.702186 18077 log.cc:824] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3: Log is configured to *not* fsync() on all Append() calls
I20260926 03:10:52.703028 18078 tablet_bootstrap.cc:492] T ad38425ea13a417ba2ede1e5582702bf P 6ae856f1f1fc48608eed00e939a33235: Bootstrap starting.
I20260926 03:10:52.705437 18077 tablet_bootstrap.cc:492] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3: No bootstrap required, opened a new log
I20260926 03:10:52.705544 18076 tablet_bootstrap.cc:492] T ad38425ea13a417ba2ede1e5582702bf P e42525cf268a4c328510560aa9934bd9: No bootstrap required, opened a new log
I20260926 03:10:52.705729 18077 ts_tablet_manager.cc:1397] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 03:10:52.705821 18076 ts_tablet_manager.cc:1397] T ad38425ea13a417ba2ede1e5582702bf P e42525cf268a4c328510560aa9934bd9: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260926 03:10:52.706907 18078 tablet_bootstrap.cc:654] T ad38425ea13a417ba2ede1e5582702bf P 6ae856f1f1fc48608eed00e939a33235: Neither blocks nor log segments found. Creating new log.
I20260926 03:10:52.708230 18078 log.cc:824] T ad38425ea13a417ba2ede1e5582702bf P 6ae856f1f1fc48608eed00e939a33235: Log is configured to *not* fsync() on all Append() calls
I20260926 03:10:52.714782 18077 raft_consensus.cc:348] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba3935f5f3040d9843dce0bd44a8fe3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46115 } } peers { permanent_uuid: "6ae856f1f1fc48608eed00e939a33235" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34767 } } peers { permanent_uuid: "e42525cf268a4c328510560aa9934bd9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36803 } }
I20260926 03:10:52.715212 18077 raft_consensus.cc:374] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:10:52.715312 18077 raft_consensus.cc:729] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba3935f5f3040d9843dce0bd44a8fe3, State: Initialized, Role: FOLLOWER
I20260926 03:10:52.716001 18077 consensus_queue.cc:260] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba3935f5f3040d9843dce0bd44a8fe3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46115 } } peers { permanent_uuid: "6ae856f1f1fc48608eed00e939a33235" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34767 } } peers { permanent_uuid: "e42525cf268a4c328510560aa9934bd9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36803 } }
I20260926 03:10:52.716984 18076 raft_consensus.cc:348] T ad38425ea13a417ba2ede1e5582702bf P e42525cf268a4c328510560aa9934bd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba3935f5f3040d9843dce0bd44a8fe3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46115 } } peers { permanent_uuid: "6ae856f1f1fc48608eed00e939a33235" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34767 } } peers { permanent_uuid: "e42525cf268a4c328510560aa9934bd9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36803 } }
I20260926 03:10:52.717562 18076 raft_consensus.cc:374] T ad38425ea13a417ba2ede1e5582702bf P e42525cf268a4c328510560aa9934bd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:10:52.717671 18076 raft_consensus.cc:729] T ad38425ea13a417ba2ede1e5582702bf P e42525cf268a4c328510560aa9934bd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e42525cf268a4c328510560aa9934bd9, State: Initialized, Role: FOLLOWER
I20260926 03:10:52.717998 18077 ts_tablet_manager.cc:1428] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260926 03:10:52.718217 18078 tablet_bootstrap.cc:492] T ad38425ea13a417ba2ede1e5582702bf P 6ae856f1f1fc48608eed00e939a33235: No bootstrap required, opened a new log
I20260926 03:10:52.718492 18078 ts_tablet_manager.cc:1397] T ad38425ea13a417ba2ede1e5582702bf P 6ae856f1f1fc48608eed00e939a33235: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20260926 03:10:52.718397 18076 consensus_queue.cc:260] T ad38425ea13a417ba2ede1e5582702bf P e42525cf268a4c328510560aa9934bd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba3935f5f3040d9843dce0bd44a8fe3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46115 } } peers { permanent_uuid: "6ae856f1f1fc48608eed00e939a33235" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34767 } } peers { permanent_uuid: "e42525cf268a4c328510560aa9934bd9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36803 } }
I20260926 03:10:52.719694 17923 heartbeater.cc:503] Master 127.0.97.62:36785 was elected leader, sending a full tablet report...
I20260926 03:10:52.720351 18076 ts_tablet_manager.cc:1428] T ad38425ea13a417ba2ede1e5582702bf P e42525cf268a4c328510560aa9934bd9: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260926 03:10:52.727720 18078 raft_consensus.cc:348] T ad38425ea13a417ba2ede1e5582702bf P 6ae856f1f1fc48608eed00e939a33235 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba3935f5f3040d9843dce0bd44a8fe3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46115 } } peers { permanent_uuid: "6ae856f1f1fc48608eed00e939a33235" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34767 } } peers { permanent_uuid: "e42525cf268a4c328510560aa9934bd9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36803 } }
I20260926 03:10:52.728157 18078 raft_consensus.cc:374] T ad38425ea13a417ba2ede1e5582702bf P 6ae856f1f1fc48608eed00e939a33235 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:10:52.728271 18078 raft_consensus.cc:729] T ad38425ea13a417ba2ede1e5582702bf P 6ae856f1f1fc48608eed00e939a33235 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ae856f1f1fc48608eed00e939a33235, State: Initialized, Role: FOLLOWER
I20260926 03:10:52.728947 18078 consensus_queue.cc:260] T ad38425ea13a417ba2ede1e5582702bf P 6ae856f1f1fc48608eed00e939a33235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba3935f5f3040d9843dce0bd44a8fe3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46115 } } peers { permanent_uuid: "6ae856f1f1fc48608eed00e939a33235" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34767 } } peers { permanent_uuid: "e42525cf268a4c328510560aa9934bd9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36803 } }
I20260926 03:10:52.730129 18057 heartbeater.cc:503] Master 127.0.97.62:36785 was elected leader, sending a full tablet report...
I20260926 03:10:52.730834 18078 ts_tablet_manager.cc:1428] T ad38425ea13a417ba2ede1e5582702bf P 6ae856f1f1fc48608eed00e939a33235: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
W20260926 03:10:52.731024 17790 tablet.cc:2369] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:10:52.759223 17924 tablet.cc:2369] T ad38425ea13a417ba2ede1e5582702bf P e42525cf268a4c328510560aa9934bd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:10:52.823215 18082 raft_consensus.cc:482] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:10:52.823606 18082 raft_consensus.cc:504] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba3935f5f3040d9843dce0bd44a8fe3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46115 } } peers { permanent_uuid: "6ae856f1f1fc48608eed00e939a33235" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34767 } } peers { permanent_uuid: "e42525cf268a4c328510560aa9934bd9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36803 } }
I20260926 03:10:52.825451 18082 leader_election.cc:288] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ae856f1f1fc48608eed00e939a33235 (127.0.97.3:34767), e42525cf268a4c328510560aa9934bd9 (127.0.97.2:36803)
W20260926 03:10:52.826723 18058 tablet.cc:2369] T ad38425ea13a417ba2ede1e5582702bf P 6ae856f1f1fc48608eed00e939a33235: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:10:52.832769 18073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:52.826048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48605 (local address 127.0.97.3:34767)
0926 03:10:52.826254 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:52.826261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:52.826945 (+ 684us) server_negotiation.cc:407] Connection header received
0926 03:10:52.827280 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:52.827284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:52.827359 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:52.827454 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:10:52.828159 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.828922 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.830967 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.831776 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.831905 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:52.832015 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:52.832436 (+ 421us) server_negotiation.cc:299] Negotiation successful
0926 03:10:52.832574 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:10:52.832755 18075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:52.825998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46199 (local address 127.0.97.2:36803)
0926 03:10:52.826220 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:52.826226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:52.827122 (+ 896us) server_negotiation.cc:407] Connection header received
0926 03:10:52.827414 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:52.827421 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:52.827496 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:52.827576 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:10:52.828246 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.828979 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.830969 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.831769 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.831913 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:52.832013 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:52.832369 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 03:10:52.832502 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 03:10:52.832840 18086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:52.825953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36803 (local address 127.0.97.1:46199)
0926 03:10:52.827005 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:52.827023 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:52.827251 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:52.827641 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:52.827645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:52.827685 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:10:52.828061 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:52.828071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.829120 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.829124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:52.830783 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:52.830799 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.832047 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.832055 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:52.832192 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:52.832218 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 03:10:52.832308 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 03:10:52.832455 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"mutex_wait_us":89,"thread_start_us":951,"threads_started":1}
I20260926 03:10:52.833439 18085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:52.825948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34767 (local address 127.0.97.1:48605)
0926 03:10:52.826828 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:52.826845 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:52.827077 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:52.827488 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:52.827493 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:52.827519 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:10:52.827998 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:52.828009 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.829059 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.829064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:52.830796 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:52.830810 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.832091 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.832096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:52.832225 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:52.832255 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 03:10:52.832966 (+ 711us) client_negotiation.cc:240] Negotiation successful
0926 03:10:52.833119 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":65,"spinlock_wait_cycles":20352,"thread_start_us":476,"threads_started":1}
I20260926 03:10:52.834038 17878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad38425ea13a417ba2ede1e5582702bf" candidate_uuid: "3ba3935f5f3040d9843dce0bd44a8fe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e42525cf268a4c328510560aa9934bd9" is_pre_election: true
I20260926 03:10:52.834448 18012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad38425ea13a417ba2ede1e5582702bf" candidate_uuid: "3ba3935f5f3040d9843dce0bd44a8fe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ae856f1f1fc48608eed00e939a33235" is_pre_election: true
I20260926 03:10:52.834609 17878 raft_consensus.cc:2454] T ad38425ea13a417ba2ede1e5582702bf P e42525cf268a4c328510560aa9934bd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ba3935f5f3040d9843dce0bd44a8fe3 in term 0.
I20260926 03:10:52.834921 18012 raft_consensus.cc:2454] T ad38425ea13a417ba2ede1e5582702bf P 6ae856f1f1fc48608eed00e939a33235 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ba3935f5f3040d9843dce0bd44a8fe3 in term 0.
I20260926 03:10:52.835311 17679 leader_election.cc:302] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba3935f5f3040d9843dce0bd44a8fe3, e42525cf268a4c328510560aa9934bd9; no voters:
I20260926 03:10:52.835731 18082 raft_consensus.cc:2781] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:10:52.835886 18082 raft_consensus.cc:482] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:10:52.835994 18082 raft_consensus.cc:3037] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:10:52.838160 18082 raft_consensus.cc:504] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba3935f5f3040d9843dce0bd44a8fe3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46115 } } peers { permanent_uuid: "6ae856f1f1fc48608eed00e939a33235" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34767 } } peers { permanent_uuid: "e42525cf268a4c328510560aa9934bd9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36803 } }
I20260926 03:10:52.838811 18082 leader_election.cc:288] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3 [CANDIDATE]: Term 1 election: Requested vote from peers 6ae856f1f1fc48608eed00e939a33235 (127.0.97.3:34767), e42525cf268a4c328510560aa9934bd9 (127.0.97.2:36803)
I20260926 03:10:52.839356 17878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad38425ea13a417ba2ede1e5582702bf" candidate_uuid: "3ba3935f5f3040d9843dce0bd44a8fe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e42525cf268a4c328510560aa9934bd9"
I20260926 03:10:52.839365 18012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad38425ea13a417ba2ede1e5582702bf" candidate_uuid: "3ba3935f5f3040d9843dce0bd44a8fe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ae856f1f1fc48608eed00e939a33235"
I20260926 03:10:52.839558 17878 raft_consensus.cc:3037] T ad38425ea13a417ba2ede1e5582702bf P e42525cf268a4c328510560aa9934bd9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:10:52.839629 18012 raft_consensus.cc:3037] T ad38425ea13a417ba2ede1e5582702bf P 6ae856f1f1fc48608eed00e939a33235 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:10:52.841785 17878 raft_consensus.cc:2454] T ad38425ea13a417ba2ede1e5582702bf P e42525cf268a4c328510560aa9934bd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba3935f5f3040d9843dce0bd44a8fe3 in term 1.
I20260926 03:10:52.841814 18012 raft_consensus.cc:2454] T ad38425ea13a417ba2ede1e5582702bf P 6ae856f1f1fc48608eed00e939a33235 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba3935f5f3040d9843dce0bd44a8fe3 in term 1.
I20260926 03:10:52.842296 17679 leader_election.cc:302] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3ba3935f5f3040d9843dce0bd44a8fe3, 6ae856f1f1fc48608eed00e939a33235, e42525cf268a4c328510560aa9934bd9; no voters:
I20260926 03:10:52.842721 18082 raft_consensus.cc:2781] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:10:52.843218 18082 raft_consensus.cc:686] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3 [term 1 LEADER]: Becoming Leader. State: Replica: 3ba3935f5f3040d9843dce0bd44a8fe3, State: Running, Role: LEADER
I20260926 03:10:52.843720 18082 consensus_queue.cc:237] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba3935f5f3040d9843dce0bd44a8fe3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46115 } } peers { permanent_uuid: "6ae856f1f1fc48608eed00e939a33235" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34767 } } peers { permanent_uuid: "e42525cf268a4c328510560aa9934bd9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36803 } }
I20260926 03:10:52.848862 17600 catalog_manager.cc:5495] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ba3935f5f3040d9843dce0bd44a8fe3 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "3ba3935f5f3040d9843dce0bd44a8fe3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ba3935f5f3040d9843dce0bd44a8fe3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ae856f1f1fc48608eed00e939a33235" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e42525cf268a4c328510560aa9934bd9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36803 } health_report { overall_health: UNKNOWN } } }
I20260926 03:10:52.909801 18071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:52.904127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59414 (local address 127.0.97.1:46115)
0926 03:10:52.904353 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:52.904358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:52.904378 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:10:52.904579 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:52.904583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:52.904666 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:52.904804 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:10:52.905192 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.905950 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.907517 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.907927 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.908047 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:52.908603 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:10:52.909177 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:10:52.909326 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:52.909433 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 03:10:52.909526 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 03:10:52.919093 18012 raft_consensus.cc:1261] T ad38425ea13a417ba2ede1e5582702bf P 6ae856f1f1fc48608eed00e939a33235 [term 1 FOLLOWER]: Refusing update from remote peer 3ba3935f5f3040d9843dce0bd44a8fe3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:10:52.919098 17878 raft_consensus.cc:1261] T ad38425ea13a417ba2ede1e5582702bf P e42525cf268a4c328510560aa9934bd9 [term 1 FOLLOWER]: Refusing update from remote peer 3ba3935f5f3040d9843dce0bd44a8fe3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:10:52.920187 18082 consensus_queue.cc:1035] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e42525cf268a4c328510560aa9934bd9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:10:52.920571 18087 consensus_queue.cc:1035] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ae856f1f1fc48608eed00e939a33235" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:10:52.948024 17599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:10:52.948365 17599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:10:52.952484 17599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be132b38c1f24693a3465b075c43a8c9: Sending DeleteTablet for 3 replicas of tablet ad38425ea13a417ba2ede1e5582702bf
I20260926 03:10:52.953786 17724 tablet_service.cc:1514] Processing DeleteTablet for tablet ad38425ea13a417ba2ede1e5582702bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:10:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59412
I20260926 03:10:52.954255 17992 tablet_service.cc:1514] Processing DeleteTablet for tablet ad38425ea13a417ba2ede1e5582702bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:10:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36918
I20260926 03:10:52.955202 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 17661
I20260926 03:10:52.956357 17858 tablet_service.cc:1514] Processing DeleteTablet for tablet ad38425ea13a417ba2ede1e5582702bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:10:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55512
I20260926 03:10:52.966079 18100 tablet_replica.cc:330] T ad38425ea13a417ba2ede1e5582702bf P 6ae856f1f1fc48608eed00e939a33235: stopping tablet replica
I20260926 03:10:52.966456 18099 tablet_replica.cc:330] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3: stopping tablet replica
I20260926 03:10:52.966673 18100 raft_consensus.cc:2229] T ad38425ea13a417ba2ede1e5582702bf P 6ae856f1f1fc48608eed00e939a33235 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:10:52.966832 18099 raft_consensus.cc:2229] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:10:52.967291 18100 raft_consensus.cc:2258] T ad38425ea13a417ba2ede1e5582702bf P 6ae856f1f1fc48608eed00e939a33235 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:10:52.967417 18099 raft_consensus.cc:2258] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:10:52.969223 18102 tablet_replica.cc:330] T ad38425ea13a417ba2ede1e5582702bf P e42525cf268a4c328510560aa9934bd9: stopping tablet replica
I20260926 03:10:52.969235 18099 ts_tablet_manager.cc:1905] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:10:52.969695 18102 raft_consensus.cc:2229] T ad38425ea13a417ba2ede1e5582702bf P e42525cf268a4c328510560aa9934bd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:10:52.970223 18102 raft_consensus.cc:2258] T ad38425ea13a417ba2ede1e5582702bf P e42525cf268a4c328510560aa9934bd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:10:52.974231 18102 ts_tablet_manager.cc:1905] T ad38425ea13a417ba2ede1e5582702bf P e42525cf268a4c328510560aa9934bd9: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 03:10:52 dist-test-slave-hmbw krb5kdc[17555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:10:52.979532 18099 ts_tablet_manager.cc:1918] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:10:52.979749 18099 log.cc:1196] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-0/wal/wals/ad38425ea13a417ba2ede1e5582702bf
I20260926 03:10:52.980306 18099 ts_tablet_manager.cc:1939] T ad38425ea13a417ba2ede1e5582702bf P 3ba3935f5f3040d9843dce0bd44a8fe3: Deleting consensus metadata
I20260926 03:10:52.982254 18102 ts_tablet_manager.cc:1918] T ad38425ea13a417ba2ede1e5582702bf P e42525cf268a4c328510560aa9934bd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:10:52.982431 18102 log.cc:1196] T ad38425ea13a417ba2ede1e5582702bf P e42525cf268a4c328510560aa9934bd9: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-1/wal/wals/ad38425ea13a417ba2ede1e5582702bf
I20260926 03:10:52.982445 17584 catalog_manager.cc:4841] TS 3ba3935f5f3040d9843dce0bd44a8fe3 (127.0.97.1:46115): tablet ad38425ea13a417ba2ede1e5582702bf (table test-table [id=3151a8e680124791bfa12ef16449aa91]) successfully deleted
I20260926 03:10:52.983006 18102 ts_tablet_manager.cc:1939] T ad38425ea13a417ba2ede1e5582702bf P e42525cf268a4c328510560aa9934bd9: Deleting consensus metadata
I20260926 03:10:52.984830 18100 ts_tablet_manager.cc:1905] T ad38425ea13a417ba2ede1e5582702bf P 6ae856f1f1fc48608eed00e939a33235: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:10:52.985611 17586 catalog_manager.cc:4841] TS e42525cf268a4c328510560aa9934bd9 (127.0.97.2:36803): tablet ad38425ea13a417ba2ede1e5582702bf (table test-table [id=3151a8e680124791bfa12ef16449aa91]) successfully deleted
I20260926 03:10:52.988945 18071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:52.957291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59424 (local address 127.0.97.1:46115)
0926 03:10:52.957476 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:52.957482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:52.963041 (+ 5559us) server_negotiation.cc:407] Connection header received
0926 03:10:52.963340 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:52.963347 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:52.963440 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:52.963530 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:52.967872 (+ 4342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.969041 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.970969 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:52.971713 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:52.971839 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:52.980454 (+ 8615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:52.980472 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:52.980491 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:10:52.980547 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:52.985003 (+ 4456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:52.986287 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:52.986301 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:52.986305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:52.986532 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:52.987019 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:52.987024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:52.987028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:52.987544 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:52.987873 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:52.988244 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 03:10:52.988522 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:10:52.989751 17784 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:10:52.991786 18100 ts_tablet_manager.cc:1918] T ad38425ea13a417ba2ede1e5582702bf P 6ae856f1f1fc48608eed00e939a33235: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:10:52.991989 18100 log.cc:1196] T ad38425ea13a417ba2ede1e5582702bf P 6ae856f1f1fc48608eed00e939a33235: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391950058021-388-0/minicluster-data/ts-2/wal/wals/ad38425ea13a417ba2ede1e5582702bf
I20260926 03:10:52.992568 18100 ts_tablet_manager.cc:1939] T ad38425ea13a417ba2ede1e5582702bf P 6ae856f1f1fc48608eed00e939a33235: Deleting consensus metadata
I20260926 03:10:52.994599 17585 catalog_manager.cc:4841] TS 6ae856f1f1fc48608eed00e939a33235 (127.0.97.3:34767): tablet ad38425ea13a417ba2ede1e5582702bf (table test-table [id=3151a8e680124791bfa12ef16449aa91]) successfully deleted
I20260926 03:10:53.089866 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 17661
I20260926 03:10:53.105969 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 17795
Sep 26 03:10:53 dist-test-slave-hmbw krb5kdc[17555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:10:53.123246 18075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:53.106953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55516 (local address 127.0.97.2:36803)
0926 03:10:53.107175 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:53.107181 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:53.107201 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:10:53.107374 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:53.107380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:53.107551 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:53.107644 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:53.109079 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:53.109907 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:53.111384 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:53.111894 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:53.112042 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:53.116694 (+ 4652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:53.116713 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:53.116730 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:10:53.116772 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:53.120062 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:53.120995 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:53.121004 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:53.121007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:53.121170 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:53.121626 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:53.121631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:53.121633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:53.122090 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:53.122337 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:53.122755 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 03:10:53.122978 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 03:10:53.124044 17918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:10:53.220424 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 17795
I20260926 03:10:53.234678 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 17929
Sep 26 03:10:53 dist-test-slave-hmbw krb5kdc[17555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:10:53.251449 18073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:53.235438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36928 (local address 127.0.97.3:34767)
0926 03:10:53.235603 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:53.235608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:53.235856 (+ 248us) server_negotiation.cc:407] Connection header received
0926 03:10:53.236027 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:53.236031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:53.236109 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:53.236207 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:53.237491 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:53.238226 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:53.239643 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:53.240101 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:53.240215 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:53.245022 (+ 4807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:53.245039 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:53.245056 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:10:53.245098 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:53.248367 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:53.249264 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:53.249272 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:53.249275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:53.249450 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:53.249818 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:53.249822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:53.249824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:53.250199 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:53.250542 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:53.250845 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 03:10:53.251072 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 03:10:53.252101 18052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:10:53.341941 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 17929
I20260926 03:10:53.355880 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 17568
I20260926 03:10:53.356927 17630 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:10:53.468426 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 17568
2026-09-26T03:10:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4374 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T03:10:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:10:53Z Disabled control of system clock
I20260926 03:10:53.541023 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40649
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:10:53.878015 18110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:10:53.878334 18110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:10:53.878439 18110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:10:53.885910 18110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:10:53.886024 18110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:10:53.886066 18110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:10:53.886142 18110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:10:53.886188 18110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:10:53.895915 18110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40649
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:10:53.897238 18110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:10:53.899114 18110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:10:53.910677 18116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:53.910633 18115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:53.910931 18118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:10:53.911365 18110 server_base.cc:1034] running on GCE node
I20260926 03:10:53.912220 18110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:10:53.913707 18110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:10:53.914921 18110 hybrid_clock.cc:648] HybridClock initialized: now 1790392253914872 us; error 38 us; skew 500 ppm
I20260926 03:10:53.915300 18110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:10:53.924353 18110 webserver.cc:460] Webserver started at http://127.0.97.62:41391/ using document root <none> and password file <none>
I20260926 03:10:53.925128 18110 fs_manager.cc:359] Metadata directory not provided
I20260926 03:10:53.925231 18110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:10:53.925550 18110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:10:53.928656 18110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "787695866aa742f69507849b3e7a09c8"
format_stamp: "Formatted at 2026-09-26 03:10:53 on dist-test-slave-hmbw"
server_key: "4d0a9408f6cb8b3074ba29a1d97335dd"
server_key_iv: "212de3b0ba7ecaab541f8d65742dd66a"
server_key_version: "encryptionkey@0"
I20260926 03:10:53.929527 18110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "787695866aa742f69507849b3e7a09c8"
format_stamp: "Formatted at 2026-09-26 03:10:53 on dist-test-slave-hmbw"
server_key: "4d0a9408f6cb8b3074ba29a1d97335dd"
server_key_iv: "212de3b0ba7ecaab541f8d65742dd66a"
server_key_version: "encryptionkey@0"
I20260926 03:10:53.935250 18110 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 03:10:53.939101 18124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:53.940589 18110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 03:10:53.940785 18110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "787695866aa742f69507849b3e7a09c8"
format_stamp: "Formatted at 2026-09-26 03:10:53 on dist-test-slave-hmbw"
server_key: "4d0a9408f6cb8b3074ba29a1d97335dd"
server_key_iv: "212de3b0ba7ecaab541f8d65742dd66a"
server_key_version: "encryptionkey@0"
I20260926 03:10:53.941015 18110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:10:53.962297 18110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:10:53.963254 18110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:10:53.963546 18110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:10:53.983476 18110 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40649
I20260926 03:10:53.983494 18175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40649 every 8 connection(s)
I20260926 03:10:53.984999 18110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:10:53.989244 18176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:10:53.995226 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 18110
I20260926 03:10:53.995385 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:10:53.995810 388 external_mini_cluster.cc:1428] Setting key 6720be22dce1a11a5e90038bf3591ff7
I20260926 03:10:54.000833 18176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 787695866aa742f69507849b3e7a09c8: Bootstrap starting.
I20260926 03:10:54.004323 18176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 787695866aa742f69507849b3e7a09c8: Neither blocks nor log segments found. Creating new log.
I20260926 03:10:54.005453 18176 log.cc:824] T 00000000000000000000000000000000 P 787695866aa742f69507849b3e7a09c8: Log is configured to *not* fsync() on all Append() calls
I20260926 03:10:54.009394 18176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 787695866aa742f69507849b3e7a09c8: No bootstrap required, opened a new log
I20260926 03:10:54.010914 18179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:53.997717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45382 (local address 127.0.97.62:40649)
0926 03:10:53.998935 (+ 1218us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:53.998948 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:53.998993 (+ 45us) server_negotiation.cc:407] Connection header received
0926 03:10:54.000832 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:54.000861 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:54.001437 (+ 576us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:54.001843 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:54.002491 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:54.003857 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:54.005406 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:54.006414 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:54.006664 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:54.007953 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:54.007962 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:54.007979 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:10:54.008692 (+ 713us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:54.008783 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:10:54.008807 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:54.008902 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:54.009256 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 03:10:54.009756 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":404,"threads_started":1}
I20260926 03:10:54.017652 18176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 787695866aa742f69507849b3e7a09c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787695866aa742f69507849b3e7a09c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40649 } }
I20260926 03:10:54.018065 18176 raft_consensus.cc:374] T 00000000000000000000000000000000 P 787695866aa742f69507849b3e7a09c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:10:54.018163 18176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 787695866aa742f69507849b3e7a09c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 787695866aa742f69507849b3e7a09c8, State: Initialized, Role: FOLLOWER
I20260926 03:10:54.018765 18176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 787695866aa742f69507849b3e7a09c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787695866aa742f69507849b3e7a09c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40649 } }
I20260926 03:10:54.019018 18176 raft_consensus.cc:388] T 00000000000000000000000000000000 P 787695866aa742f69507849b3e7a09c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:10:54.019145 18176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 787695866aa742f69507849b3e7a09c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:10:54.019301 18176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 787695866aa742f69507849b3e7a09c8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:10:54.021621 18176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 787695866aa742f69507849b3e7a09c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787695866aa742f69507849b3e7a09c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40649 } }
I20260926 03:10:54.022197 18176 leader_election.cc:302] T 00000000000000000000000000000000 P 787695866aa742f69507849b3e7a09c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 787695866aa742f69507849b3e7a09c8; no voters:
I20260926 03:10:54.022843 18176 leader_election.cc:288] T 00000000000000000000000000000000 P 787695866aa742f69507849b3e7a09c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:10:54.023053 18181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 787695866aa742f69507849b3e7a09c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:10:54.024173 18181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 787695866aa742f69507849b3e7a09c8 [term 1 LEADER]: Becoming Leader. State: Replica: 787695866aa742f69507849b3e7a09c8, State: Running, Role: LEADER
I20260926 03:10:54.024771 18181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 787695866aa742f69507849b3e7a09c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787695866aa742f69507849b3e7a09c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40649 } }
I20260926 03:10:54.025174 18176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 787695866aa742f69507849b3e7a09c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:10:54.029453 18183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 787695866aa742f69507849b3e7a09c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 787695866aa742f69507849b3e7a09c8. Latest consensus state: current_term: 1 leader_uuid: "787695866aa742f69507849b3e7a09c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787695866aa742f69507849b3e7a09c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40649 } } }
I20260926 03:10:54.029613 18182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 787695866aa742f69507849b3e7a09c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "787695866aa742f69507849b3e7a09c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787695866aa742f69507849b3e7a09c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40649 } } }
I20260926 03:10:54.029959 18182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 787695866aa742f69507849b3e7a09c8 [sys.catalog]: This master's current role is: LEADER
I20260926 03:10:54.029953 18183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 787695866aa742f69507849b3e7a09c8 [sys.catalog]: This master's current role is: LEADER
I20260926 03:10:54.031415 18191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:10:54.035971 18191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:10:54.045575 18191 catalog_manager.cc:1347] Generated new cluster ID: 78806c8f63a54cf7b346e2252f72236c
I20260926 03:10:54.045817 18191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:10:54.056756 18191 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:10:54.058668 18191 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:10:54.071521 18191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 787695866aa742f69507849b3e7a09c8: Generated new TSK 0
I20260926 03:10:54.072412 18191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:10:54.085047 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40649
--builtin_ntp_servers=127.0.97.20:39845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:10:54.418818 18200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:10:54.419104 18200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:10:54.419216 18200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:10:54.426695 18200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:10:54.426816 18200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:10:54.426962 18200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:10:54.436514 18200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:10:54.437824 18200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:10:54.439617 18200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:10:54.452430 18206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:54.452394 18205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:54.452469 18208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:10:54.455952 18200 server_base.cc:1034] running on GCE node
I20260926 03:10:54.456583 18200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:10:54.457572 18200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:10:54.458770 18200 hybrid_clock.cc:648] HybridClock initialized: now 1790392254458717 us; error 43 us; skew 500 ppm
I20260926 03:10:54.459103 18200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:10:54.461359 18200 webserver.cc:460] Webserver started at http://127.0.97.1:35705/ using document root <none> and password file <none>
I20260926 03:10:54.462046 18200 fs_manager.cc:359] Metadata directory not provided
I20260926 03:10:54.462136 18200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:10:54.462435 18200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:10:54.465389 18200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-0/data/instance:
uuid: "058f531272aa484fbcd67affbeafce0c"
format_stamp: "Formatted at 2026-09-26 03:10:54 on dist-test-slave-hmbw"
server_key: "034bf9074248de556cb5543a868b8459"
server_key_iv: "7d3b8adeec3c7b43c1a091794a5f2511"
server_key_version: "encryptionkey@0"
I20260926 03:10:54.466215 18200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-0/wal/instance:
uuid: "058f531272aa484fbcd67affbeafce0c"
format_stamp: "Formatted at 2026-09-26 03:10:54 on dist-test-slave-hmbw"
server_key: "034bf9074248de556cb5543a868b8459"
server_key_iv: "7d3b8adeec3c7b43c1a091794a5f2511"
server_key_version: "encryptionkey@0"
I20260926 03:10:54.471714 18200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 03:10:54.475541 18214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:54.476969 18200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:10:54.477130 18200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "058f531272aa484fbcd67affbeafce0c"
format_stamp: "Formatted at 2026-09-26 03:10:54 on dist-test-slave-hmbw"
server_key: "034bf9074248de556cb5543a868b8459"
server_key_iv: "7d3b8adeec3c7b43c1a091794a5f2511"
server_key_version: "encryptionkey@0"
I20260926 03:10:54.477347 18200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:10:54.494421 18200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:10:54.495389 18200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:10:54.495653 18200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:10:54.496636 18200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:10:54.498134 18200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:10:54.498232 18200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:54.498338 18200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:10:54.498407 18200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:54.529620 18200 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34955
I20260926 03:10:54.529675 18326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34955 every 8 connection(s)
I20260926 03:10:54.531111 18200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:10:54.539230 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 18200
I20260926 03:10:54.539433 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:10:54.539889 388 external_mini_cluster.cc:1428] Setting key 2961d32d6862f47f469f7e10aca1ae73
I20260926 03:10:54.542840 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40649
--builtin_ntp_servers=127.0.97.20:39845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:10:54.544281 18329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:54.533649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41501 (local address 127.0.97.62:40649)
0926 03:10:54.534093 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:54.534106 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:54.535259 (+ 1153us) server_negotiation.cc:407] Connection header received
0926 03:10:54.537101 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:54.537108 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:54.537204 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:54.537302 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:54.538417 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:54.539480 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:54.541191 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:54.541765 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:54.541902 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:54.543487 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:54.543494 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:54.543500 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:10:54.543523 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:54.543573 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:10:54.543590 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:54.543685 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:54.543863 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 03:10:54.543963 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":239,"threads_started":1}
I20260926 03:10:54.545394 18330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:54.534112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40649 (local address 127.0.97.1:41501)
0926 03:10:54.535084 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:54.535128 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:54.536815 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:54.537555 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:54.537564 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:54.537653 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:10:54.538242 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:54.538260 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:54.539666 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:54.539671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:54.541041 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:54.541056 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:54.542062 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:54.542071 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:54.542381 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:54.543151 (+ 770us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:10:54.543154 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:10:54.543201 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:10:54.543285 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:10:54.543683 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:10:54.543690 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:10:54.543697 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 03:10:54.543876 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 03:10:54.544205 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":344,"threads_started":1}
I20260926 03:10:54.546923 18327 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40649
I20260926 03:10:54.547211 18327 heartbeater.cc:464] Registering TS with master...
I20260926 03:10:54.547978 18327 heartbeater.cc:511] Master 127.0.97.62:40649 requested a full tablet report, sending...
I20260926 03:10:54.549891 18141 ts_manager.cc:194] Registered new tserver with Master: 058f531272aa484fbcd67affbeafce0c (127.0.97.1:34955)
I20260926 03:10:54.551502 18141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:41501
W20260926 03:10:54.866350 18331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:10:54.866722 18331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:10:54.866868 18331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:10:54.874346 18331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:10:54.874470 18331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:10:54.874639 18331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:10:54.884439 18331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:10:54.885739 18331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:10:54.887555 18331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:10:54.900698 18336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:54.900733 18339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:54.900760 18337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:10:54.901443 18331 server_base.cc:1034] running on GCE node
I20260926 03:10:54.902064 18331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:10:54.903131 18331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:10:54.904419 18331 hybrid_clock.cc:648] HybridClock initialized: now 1790392254904370 us; error 45 us; skew 500 ppm
I20260926 03:10:54.904774 18331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:10:54.907200 18331 webserver.cc:460] Webserver started at http://127.0.97.2:43353/ using document root <none> and password file <none>
I20260926 03:10:54.907950 18331 fs_manager.cc:359] Metadata directory not provided
I20260926 03:10:54.908037 18331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:10:54.908316 18331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:10:54.911294 18331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-1/data/instance:
uuid: "d9a1d643d3004ec0b48ef3f8d8d11c00"
format_stamp: "Formatted at 2026-09-26 03:10:54 on dist-test-slave-hmbw"
server_key: "3ea5c569d2926c19bf4cd4f641d98246"
server_key_iv: "64d4dd1ceac7a6aaf50b5657a79b52e8"
server_key_version: "encryptionkey@0"
I20260926 03:10:54.912071 18331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d9a1d643d3004ec0b48ef3f8d8d11c00"
format_stamp: "Formatted at 2026-09-26 03:10:54 on dist-test-slave-hmbw"
server_key: "3ea5c569d2926c19bf4cd4f641d98246"
server_key_iv: "64d4dd1ceac7a6aaf50b5657a79b52e8"
server_key_version: "encryptionkey@0"
I20260926 03:10:54.917454 18331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:10:54.921098 18345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:54.922626 18331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 03:10:54.922777 18331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "d9a1d643d3004ec0b48ef3f8d8d11c00"
format_stamp: "Formatted at 2026-09-26 03:10:54 on dist-test-slave-hmbw"
server_key: "3ea5c569d2926c19bf4cd4f641d98246"
server_key_iv: "64d4dd1ceac7a6aaf50b5657a79b52e8"
server_key_version: "encryptionkey@0"
I20260926 03:10:54.922994 18331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:10:54.943106 18331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:10:54.944000 18331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:10:54.944289 18331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:10:54.945288 18331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:10:54.946913 18331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:10:54.947012 18331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:54.947104 18331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:10:54.947158 18331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:54.978621 18331 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34811
I20260926 03:10:54.978713 18457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34811 every 8 connection(s)
I20260926 03:10:54.979864 18331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:10:54.986407 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 18331
I20260926 03:10:54.986599 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:10:54.987097 388 external_mini_cluster.cc:1428] Setting key 148fef43f8b846339566fedc6bf3a86c
I20260926 03:10:54.989979 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40649
--builtin_ntp_servers=127.0.97.20:39845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:10:54.999126 18329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:54.982669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48151 (local address 127.0.97.62:40649)
0926 03:10:54.990661 (+ 7992us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:54.990666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:54.990684 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:10:54.990736 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:54.990740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:54.990804 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:54.990907 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:54.992458 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:54.993446 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:54.995856 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:54.996442 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:54.996581 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:54.998268 (+ 1687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:54.998275 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:54.998279 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:10:54.998303 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:54.998344 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:10:54.998360 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:54.998443 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:54.998682 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 03:10:54.998798 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7823}
I20260926 03:10:55.000712 18460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:54.983289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40649 (local address 127.0.97.2:48151)
0926 03:10:54.984512 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:54.984555 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:54.986881 (+ 2326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:54.991273 (+ 4392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:54.991288 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:54.991446 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:10:54.992232 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:54.992257 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:54.993636 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:54.993641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:54.995674 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:54.995692 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:54.996713 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:54.996721 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:54.997022 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:54.997858 (+ 836us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:10:54.997861 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:10:54.997917 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:10:54.998035 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:10:54.998450 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:10:54.998458 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:10:54.998466 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 03:10:54.998671 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 03:10:54.999113 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":380,"threads_started":1}
I20260926 03:10:55.002322 18458 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40649
I20260926 03:10:55.002655 18458 heartbeater.cc:464] Registering TS with master...
I20260926 03:10:55.003448 18458 heartbeater.cc:511] Master 127.0.97.62:40649 requested a full tablet report, sending...
I20260926 03:10:55.005026 18141 ts_manager.cc:194] Registered new tserver with Master: d9a1d643d3004ec0b48ef3f8d8d11c00 (127.0.97.2:34811)
I20260926 03:10:55.005856 18141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:48151
W20260926 03:10:55.329617 18461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:10:55.329905 18461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:10:55.330014 18461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:10:55.337541 18461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:10:55.337661 18461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:10:55.337797 18461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:10:55.347623 18461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:10:55.348933 18461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:10:55.350730 18461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:10:55.363768 18466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:55.363809 18467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:55.363833 18469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:10:55.364660 18461 server_base.cc:1034] running on GCE node
I20260926 03:10:55.365288 18461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:10:55.366333 18461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:10:55.367604 18461 hybrid_clock.cc:648] HybridClock initialized: now 1790392255367546 us; error 78 us; skew 500 ppm
I20260926 03:10:55.367996 18461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:10:55.370635 18461 webserver.cc:460] Webserver started at http://127.0.97.3:38993/ using document root <none> and password file <none>
I20260926 03:10:55.371382 18461 fs_manager.cc:359] Metadata directory not provided
I20260926 03:10:55.371490 18461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:10:55.371795 18461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:10:55.374895 18461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-2/data/instance:
uuid: "e95f5e34e1b6473d82783d1f36a2326b"
format_stamp: "Formatted at 2026-09-26 03:10:55 on dist-test-slave-hmbw"
server_key: "1c33fb7894b32beaf513f22d8112694a"
server_key_iv: "adffefc3218e541bef7cced17baeab65"
server_key_version: "encryptionkey@0"
I20260926 03:10:55.375777 18461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e95f5e34e1b6473d82783d1f36a2326b"
format_stamp: "Formatted at 2026-09-26 03:10:55 on dist-test-slave-hmbw"
server_key: "1c33fb7894b32beaf513f22d8112694a"
server_key_iv: "adffefc3218e541bef7cced17baeab65"
server_key_version: "encryptionkey@0"
I20260926 03:10:55.381623 18461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:10:55.385718 18475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:55.387310 18461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:10:55.387488 18461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "e95f5e34e1b6473d82783d1f36a2326b"
format_stamp: "Formatted at 2026-09-26 03:10:55 on dist-test-slave-hmbw"
server_key: "1c33fb7894b32beaf513f22d8112694a"
server_key_iv: "adffefc3218e541bef7cced17baeab65"
server_key_version: "encryptionkey@0"
I20260926 03:10:55.387717 18461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:10:55.414155 18461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:10:55.415114 18461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:10:55.415418 18461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:10:55.416489 18461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:10:55.418051 18461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:10:55.418144 18461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:55.418234 18461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:10:55.418289 18461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:55.449571 18461 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35009
I20260926 03:10:55.449640 18587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35009 every 8 connection(s)
I20260926 03:10:55.450893 18461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:10:55.456177 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 18461
I20260926 03:10:55.456333 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:10:55.456820 388 external_mini_cluster.cc:1428] Setting key 3619d152be9901c0df39d807ab384360
I20260926 03:10:55.463833 18329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:55.453558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43431 (local address 127.0.97.62:40649)
0926 03:10:55.453735 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:55.453740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:55.455156 (+ 1416us) server_negotiation.cc:407] Connection header received
0926 03:10:55.457058 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:55.457065 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:55.457159 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:55.457276 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:55.458729 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.459653 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.461282 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.461675 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.461770 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:55.463145 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:55.463150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:55.463153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:10:55.463173 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:55.463205 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:10:55.463216 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:55.463286 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:55.463494 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 03:10:55.463604 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:10:55.464874 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:55.454001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40649 (local address 127.0.97.3:43431)
0926 03:10:55.454993 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:55.455035 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:55.456760 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:55.457600 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:55.457614 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:55.457730 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:10:55.458544 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:55.458571 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.459832 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.459838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:55.461155 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:55.461167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.461845 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.461850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:55.462087 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:55.462798 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:10:55.462801 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:10:55.462842 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:10:55.462956 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:10:55.463295 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:10:55.463302 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:10:55.463307 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:10:55.463466 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 03:10:55.463782 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":324,"threads_started":1}
I20260926 03:10:55.466454 18588 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40649
I20260926 03:10:55.466810 18588 heartbeater.cc:464] Registering TS with master...
I20260926 03:10:55.467656 18588 heartbeater.cc:511] Master 127.0.97.62:40649 requested a full tablet report, sending...
I20260926 03:10:55.469259 18141 ts_manager.cc:194] Registered new tserver with Master: e95f5e34e1b6473d82783d1f36a2326b (127.0.97.3:35009)
I20260926 03:10:55.470080 18141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:43431
I20260926 03:10:55.473071 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:10:55.484098 18329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:55.477443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60544 (local address 127.0.97.62:40649)
0926 03:10:55.477616 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:55.477621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:55.477927 (+ 306us) server_negotiation.cc:407] Connection header received
0926 03:10:55.478115 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:55.478120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:55.478209 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:55.478562 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:55.478945 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.479730 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.481653 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.482099 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.482247 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:55.483395 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:55.483401 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:55.483404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:10:55.483426 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:55.483463 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:10:55.483476 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:55.483555 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:55.483687 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 03:10:55.483793 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:10:55.488493 18141 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:10:55.495831 18141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:10:55.521930 18601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:55.509129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42162 (local address 127.0.97.1:34955)
0926 03:10:55.509882 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:55.509903 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:55.510678 (+ 775us) server_negotiation.cc:407] Connection header received
0926 03:10:55.511414 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:55.511424 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:55.511668 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:55.511830 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:10:55.512867 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.514784 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.518121 (+ 3337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.519921 (+ 1800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.520083 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:55.520412 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:55.521277 (+ 865us) server_negotiation.cc:299] Negotiation successful
0926 03:10:55.521510 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":418,"threads_started":1}
I20260926 03:10:55.522878 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:55.508909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34955 (local address 127.0.0.1:42162)
0926 03:10:55.510520 (+ 1611us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:55.510571 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:55.511197 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:55.512168 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:55.512177 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:55.512211 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:10:55.512655 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:55.512668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.515049 (+ 2381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.515056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:55.517758 (+ 2702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:55.517779 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.520654 (+ 2875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.520663 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:55.520935 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:55.521005 (+ 70us) client_negotiation.cc:769] Sending connection context
0926 03:10:55.522202 (+ 1197us) client_negotiation.cc:240] Negotiation successful
0926 03:10:55.522522 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1315,"thread_start_us":1833,"threads_started":1}
I20260926 03:10:55.525287 18600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:55.509409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35009 (local address 127.0.0.1:43060)
0926 03:10:55.510751 (+ 1342us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:55.510766 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:55.510959 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:55.513401 (+ 2442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:55.513406 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:55.513437 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:10:55.513887 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:55.513899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.517760 (+ 3861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.517766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:55.520388 (+ 2622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:55.522373 (+ 1985us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.524430 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.524438 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:55.524649 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:55.524683 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 03:10:55.524778 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 03:10:55.524947 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"thread_start_us":651,"threads_started":1}
I20260926 03:10:55.525486 18603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:55.510024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43060 (local address 127.0.97.3:35009)
0926 03:10:55.510856 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:55.510862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:55.510890 (+ 28us) server_negotiation.cc:407] Connection header received
0926 03:10:55.511186 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:55.511194 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:55.511460 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:55.511673 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:10:55.514164 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.517556 (+ 3392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.522720 (+ 5164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.524197 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.524351 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:55.524645 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:55.524836 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 03:10:55.525002 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":542,"threads_started":1}
I20260926 03:10:55.529407 18523 tablet_service.cc:1467] Processing CreateTablet for tablet ea9519dd5bf340ab9bb47ed576eb9589 (DEFAULT_TABLE table=test-table [id=f20c71ae25db46d49f4f89030bc362eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:10:55.529410 18262 tablet_service.cc:1467] Processing CreateTablet for tablet ea9519dd5bf340ab9bb47ed576eb9589 (DEFAULT_TABLE table=test-table [id=f20c71ae25db46d49f4f89030bc362eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:10:55.531142 18523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea9519dd5bf340ab9bb47ed576eb9589. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:10:55.531144 18262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea9519dd5bf340ab9bb47ed576eb9589. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:10:55.531735 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:55.511147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34811 (local address 127.0.0.1:60570)
0926 03:10:55.523167 (+ 12020us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:55.523190 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:55.523314 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:55.524992 (+ 1678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:55.524997 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:55.525025 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:10:55.525221 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:55.525232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.527338 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.527343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:55.528779 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:55.528795 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.530940 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.530947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:55.531165 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:55.531206 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 03:10:55.531271 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 03:10:55.531426 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11882,"thread_start_us":13587,"threads_started":1}
I20260926 03:10:55.532850 18605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:55.511386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60570 (local address 127.0.97.2:34811)
0926 03:10:55.523904 (+ 12518us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:55.523915 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:55.523950 (+ 35us) server_negotiation.cc:407] Connection header received
0926 03:10:55.524074 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:55.524084 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:55.524415 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:55.525418 (+ 1003us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:10:55.525660 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.527159 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.529001 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.530708 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.532010 (+ 1302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:55.532269 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:55.532343 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 03:10:55.532514 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12325,"thread_start_us":12170,"threads_started":1}
I20260926 03:10:55.537626 18393 tablet_service.cc:1467] Processing CreateTablet for tablet ea9519dd5bf340ab9bb47ed576eb9589 (DEFAULT_TABLE table=test-table [id=f20c71ae25db46d49f4f89030bc362eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:10:55.539299 18393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea9519dd5bf340ab9bb47ed576eb9589. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:10:55.551666 18606 tablet_bootstrap.cc:492] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b: Bootstrap starting.
I20260926 03:10:55.554596 18327 heartbeater.cc:503] Master 127.0.97.62:40649 was elected leader, sending a full tablet report...
I20260926 03:10:55.556114 18606 tablet_bootstrap.cc:654] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b: Neither blocks nor log segments found. Creating new log.
I20260926 03:10:55.556996 18608 tablet_bootstrap.cc:492] T ea9519dd5bf340ab9bb47ed576eb9589 P d9a1d643d3004ec0b48ef3f8d8d11c00: Bootstrap starting.
I20260926 03:10:55.554620 18607 tablet_bootstrap.cc:492] T ea9519dd5bf340ab9bb47ed576eb9589 P 058f531272aa484fbcd67affbeafce0c: Bootstrap starting.
I20260926 03:10:55.557866 18606 log.cc:824] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b: Log is configured to *not* fsync() on all Append() calls
I20260926 03:10:55.560958 18607 tablet_bootstrap.cc:654] T ea9519dd5bf340ab9bb47ed576eb9589 P 058f531272aa484fbcd67affbeafce0c: Neither blocks nor log segments found. Creating new log.
I20260926 03:10:55.562239 18606 tablet_bootstrap.cc:492] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b: No bootstrap required, opened a new log
I20260926 03:10:55.562230 18607 log.cc:824] T ea9519dd5bf340ab9bb47ed576eb9589 P 058f531272aa484fbcd67affbeafce0c: Log is configured to *not* fsync() on all Append() calls
I20260926 03:10:55.562314 18608 tablet_bootstrap.cc:654] T ea9519dd5bf340ab9bb47ed576eb9589 P d9a1d643d3004ec0b48ef3f8d8d11c00: Neither blocks nor log segments found. Creating new log.
I20260926 03:10:55.562565 18606 ts_tablet_manager.cc:1397] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.002s
I20260926 03:10:55.563513 18608 log.cc:824] T ea9519dd5bf340ab9bb47ed576eb9589 P d9a1d643d3004ec0b48ef3f8d8d11c00: Log is configured to *not* fsync() on all Append() calls
I20260926 03:10:55.565397 18607 tablet_bootstrap.cc:492] T ea9519dd5bf340ab9bb47ed576eb9589 P 058f531272aa484fbcd67affbeafce0c: No bootstrap required, opened a new log
I20260926 03:10:55.565689 18607 ts_tablet_manager.cc:1397] T ea9519dd5bf340ab9bb47ed576eb9589 P 058f531272aa484fbcd67affbeafce0c: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260926 03:10:55.565987 18608 tablet_bootstrap.cc:492] T ea9519dd5bf340ab9bb47ed576eb9589 P d9a1d643d3004ec0b48ef3f8d8d11c00: No bootstrap required, opened a new log
I20260926 03:10:55.566237 18608 ts_tablet_manager.cc:1397] T ea9519dd5bf340ab9bb47ed576eb9589 P d9a1d643d3004ec0b48ef3f8d8d11c00: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 03:10:55.573145 18606 raft_consensus.cc:348] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058f531272aa484fbcd67affbeafce0c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34955 } } peers { permanent_uuid: "e95f5e34e1b6473d82783d1f36a2326b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35009 } } peers { permanent_uuid: "d9a1d643d3004ec0b48ef3f8d8d11c00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34811 } }
I20260926 03:10:55.573683 18606 raft_consensus.cc:374] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:10:55.573432 18608 raft_consensus.cc:348] T ea9519dd5bf340ab9bb47ed576eb9589 P d9a1d643d3004ec0b48ef3f8d8d11c00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058f531272aa484fbcd67affbeafce0c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34955 } } peers { permanent_uuid: "e95f5e34e1b6473d82783d1f36a2326b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35009 } } peers { permanent_uuid: "d9a1d643d3004ec0b48ef3f8d8d11c00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34811 } }
I20260926 03:10:55.573792 18606 raft_consensus.cc:729] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e95f5e34e1b6473d82783d1f36a2326b, State: Initialized, Role: FOLLOWER
I20260926 03:10:55.573884 18608 raft_consensus.cc:374] T ea9519dd5bf340ab9bb47ed576eb9589 P d9a1d643d3004ec0b48ef3f8d8d11c00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:10:55.573989 18608 raft_consensus.cc:729] T ea9519dd5bf340ab9bb47ed576eb9589 P d9a1d643d3004ec0b48ef3f8d8d11c00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9a1d643d3004ec0b48ef3f8d8d11c00, State: Initialized, Role: FOLLOWER
I20260926 03:10:55.574548 18606 consensus_queue.cc:260] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058f531272aa484fbcd67affbeafce0c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34955 } } peers { permanent_uuid: "e95f5e34e1b6473d82783d1f36a2326b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35009 } } peers { permanent_uuid: "d9a1d643d3004ec0b48ef3f8d8d11c00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34811 } }
I20260926 03:10:55.574863 18608 consensus_queue.cc:260] T ea9519dd5bf340ab9bb47ed576eb9589 P d9a1d643d3004ec0b48ef3f8d8d11c00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058f531272aa484fbcd67affbeafce0c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34955 } } peers { permanent_uuid: "e95f5e34e1b6473d82783d1f36a2326b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35009 } } peers { permanent_uuid: "d9a1d643d3004ec0b48ef3f8d8d11c00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34811 } }
I20260926 03:10:55.576789 18458 heartbeater.cc:503] Master 127.0.97.62:40649 was elected leader, sending a full tablet report...
I20260926 03:10:55.576680 18607 raft_consensus.cc:348] T ea9519dd5bf340ab9bb47ed576eb9589 P 058f531272aa484fbcd67affbeafce0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058f531272aa484fbcd67affbeafce0c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34955 } } peers { permanent_uuid: "e95f5e34e1b6473d82783d1f36a2326b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35009 } } peers { permanent_uuid: "d9a1d643d3004ec0b48ef3f8d8d11c00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34811 } }
I20260926 03:10:55.577099 18606 ts_tablet_manager.cc:1428] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b: Time spent starting tablet: real 0.014s user 0.006s sys 0.008s
I20260926 03:10:55.577147 18607 raft_consensus.cc:374] T ea9519dd5bf340ab9bb47ed576eb9589 P 058f531272aa484fbcd67affbeafce0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:10:55.577230 18607 raft_consensus.cc:729] T ea9519dd5bf340ab9bb47ed576eb9589 P 058f531272aa484fbcd67affbeafce0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 058f531272aa484fbcd67affbeafce0c, State: Initialized, Role: FOLLOWER
I20260926 03:10:55.577378 18608 ts_tablet_manager.cc:1428] T ea9519dd5bf340ab9bb47ed576eb9589 P d9a1d643d3004ec0b48ef3f8d8d11c00: Time spent starting tablet: real 0.011s user 0.003s sys 0.009s
I20260926 03:10:55.577680 18588 heartbeater.cc:503] Master 127.0.97.62:40649 was elected leader, sending a full tablet report...
I20260926 03:10:55.577847 18607 consensus_queue.cc:260] T ea9519dd5bf340ab9bb47ed576eb9589 P 058f531272aa484fbcd67affbeafce0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058f531272aa484fbcd67affbeafce0c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34955 } } peers { permanent_uuid: "e95f5e34e1b6473d82783d1f36a2326b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35009 } } peers { permanent_uuid: "d9a1d643d3004ec0b48ef3f8d8d11c00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34811 } }
I20260926 03:10:55.580462 18607 ts_tablet_manager.cc:1428] T ea9519dd5bf340ab9bb47ed576eb9589 P 058f531272aa484fbcd67affbeafce0c: Time spent starting tablet: real 0.015s user 0.017s sys 0.000s
I20260926 03:10:55.641355 18612 raft_consensus.cc:482] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:10:55.641667 18612 raft_consensus.cc:504] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058f531272aa484fbcd67affbeafce0c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34955 } } peers { permanent_uuid: "e95f5e34e1b6473d82783d1f36a2326b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35009 } } peers { permanent_uuid: "d9a1d643d3004ec0b48ef3f8d8d11c00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34811 } }
I20260926 03:10:55.643492 18612 leader_election.cc:288] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 058f531272aa484fbcd67affbeafce0c (127.0.97.1:34955), d9a1d643d3004ec0b48ef3f8d8d11c00 (127.0.97.2:34811)
I20260926 03:10:55.650291 18601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:55.643928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45897 (local address 127.0.97.1:34955)
0926 03:10:55.644145 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:55.644151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:55.644173 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:10:55.644382 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:55.644387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:55.644472 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:55.644564 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:10:55.645173 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.646066 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.648188 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.649416 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.649514 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:55.649586 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:55.649970 (+ 384us) server_negotiation.cc:299] Negotiation successful
0926 03:10:55.650085 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:10:55.650388 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:55.643786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34955 (local address 127.0.97.3:45897)
0926 03:10:55.644000 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:55.644022 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:55.644157 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:55.644658 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:55.644670 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:55.644711 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:10:55.644988 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:55.645001 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.646191 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.646196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:55.647986 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:55.648000 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.649579 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.649586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:55.649769 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:55.649808 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 03:10:55.649910 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 03:10:55.650065 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":46}
I20260926 03:10:55.651476 18615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:55.644170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34811 (local address 127.0.97.3:35417)
0926 03:10:55.645013 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:55.645030 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:55.645142 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:55.645463 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:55.645467 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:55.645489 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:10:55.645796 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:55.645804 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.647181 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.647185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:55.649061 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:55.649079 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.650707 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.650713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:55.650854 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:55.650911 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 03:10:55.651001 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 03:10:55.651140 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"mutex_wait_us":4,"thread_start_us":445,"threads_started":1}
I20260926 03:10:55.651508 18605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:55.644446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35417 (local address 127.0.97.2:34811)
0926 03:10:55.644614 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:55.644620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:55.645099 (+ 479us) server_negotiation.cc:407] Connection header received
0926 03:10:55.645256 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:55.645260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:55.645350 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:55.645430 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:10:55.645953 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.646975 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.649269 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.650388 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.650545 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:55.650637 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:55.651058 (+ 421us) server_negotiation.cc:299] Negotiation successful
0926 03:10:55.651203 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 03:10:55.651455 18282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea9519dd5bf340ab9bb47ed576eb9589" candidate_uuid: "e95f5e34e1b6473d82783d1f36a2326b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "058f531272aa484fbcd67affbeafce0c" is_pre_election: true
I20260926 03:10:55.652022 18282 raft_consensus.cc:2454] T ea9519dd5bf340ab9bb47ed576eb9589 P 058f531272aa484fbcd67affbeafce0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e95f5e34e1b6473d82783d1f36a2326b in term 0.
I20260926 03:10:55.652575 18413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea9519dd5bf340ab9bb47ed576eb9589" candidate_uuid: "e95f5e34e1b6473d82783d1f36a2326b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9a1d643d3004ec0b48ef3f8d8d11c00" is_pre_election: true
I20260926 03:10:55.652750 18478 leader_election.cc:302] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 058f531272aa484fbcd67affbeafce0c, e95f5e34e1b6473d82783d1f36a2326b; no voters:
I20260926 03:10:55.653014 18413 raft_consensus.cc:2454] T ea9519dd5bf340ab9bb47ed576eb9589 P d9a1d643d3004ec0b48ef3f8d8d11c00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e95f5e34e1b6473d82783d1f36a2326b in term 0.
I20260926 03:10:55.653157 18612 raft_consensus.cc:2781] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:10:55.653301 18612 raft_consensus.cc:482] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:10:55.653414 18612 raft_consensus.cc:3037] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:10:55.655669 18612 raft_consensus.cc:504] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058f531272aa484fbcd67affbeafce0c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34955 } } peers { permanent_uuid: "e95f5e34e1b6473d82783d1f36a2326b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35009 } } peers { permanent_uuid: "d9a1d643d3004ec0b48ef3f8d8d11c00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34811 } }
I20260926 03:10:55.656208 18612 leader_election.cc:288] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b [CANDIDATE]: Term 1 election: Requested vote from peers 058f531272aa484fbcd67affbeafce0c (127.0.97.1:34955), d9a1d643d3004ec0b48ef3f8d8d11c00 (127.0.97.2:34811)
I20260926 03:10:55.656719 18282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea9519dd5bf340ab9bb47ed576eb9589" candidate_uuid: "e95f5e34e1b6473d82783d1f36a2326b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "058f531272aa484fbcd67affbeafce0c"
I20260926 03:10:55.656759 18413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea9519dd5bf340ab9bb47ed576eb9589" candidate_uuid: "e95f5e34e1b6473d82783d1f36a2326b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9a1d643d3004ec0b48ef3f8d8d11c00"
I20260926 03:10:55.656951 18282 raft_consensus.cc:3037] T ea9519dd5bf340ab9bb47ed576eb9589 P 058f531272aa484fbcd67affbeafce0c [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:10:55.656953 18413 raft_consensus.cc:3037] T ea9519dd5bf340ab9bb47ed576eb9589 P d9a1d643d3004ec0b48ef3f8d8d11c00 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:10:55.660270 18413 raft_consensus.cc:2454] T ea9519dd5bf340ab9bb47ed576eb9589 P d9a1d643d3004ec0b48ef3f8d8d11c00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e95f5e34e1b6473d82783d1f36a2326b in term 1.
I20260926 03:10:55.660270 18282 raft_consensus.cc:2454] T ea9519dd5bf340ab9bb47ed576eb9589 P 058f531272aa484fbcd67affbeafce0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e95f5e34e1b6473d82783d1f36a2326b in term 1.
I20260926 03:10:55.660876 18478 leader_election.cc:302] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 058f531272aa484fbcd67affbeafce0c, e95f5e34e1b6473d82783d1f36a2326b; no voters:
I20260926 03:10:55.661216 18612 raft_consensus.cc:2781] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:10:55.661684 18612 raft_consensus.cc:686] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b [term 1 LEADER]: Becoming Leader. State: Replica: e95f5e34e1b6473d82783d1f36a2326b, State: Running, Role: LEADER
I20260926 03:10:55.662184 18612 consensus_queue.cc:237] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058f531272aa484fbcd67affbeafce0c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34955 } } peers { permanent_uuid: "e95f5e34e1b6473d82783d1f36a2326b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35009 } } peers { permanent_uuid: "d9a1d643d3004ec0b48ef3f8d8d11c00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34811 } }
I20260926 03:10:55.666962 18141 catalog_manager.cc:5495] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b reported cstate change: term changed from 0 to 1, leader changed from <none> to e95f5e34e1b6473d82783d1f36a2326b (127.0.97.3). New cstate: current_term: 1 leader_uuid: "e95f5e34e1b6473d82783d1f36a2326b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "058f531272aa484fbcd67affbeafce0c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e95f5e34e1b6473d82783d1f36a2326b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9a1d643d3004ec0b48ef3f8d8d11c00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34811 } health_report { overall_health: UNKNOWN } } }
W20260926 03:10:55.702651 18589 tablet.cc:2369] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:10:55.716363 18603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:55.710632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43072 (local address 127.0.97.3:35009)
0926 03:10:55.710789 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:55.710794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:55.710819 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:10:55.710910 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:55.710913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:55.710988 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:55.711124 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:10:55.711560 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.712290 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.713880 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.714398 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.714554 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:55.715176 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:10:55.715765 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:10:55.715906 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:55.716025 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 03:10:55.716101 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 03:10:55.725097 18413 raft_consensus.cc:1261] T ea9519dd5bf340ab9bb47ed576eb9589 P d9a1d643d3004ec0b48ef3f8d8d11c00 [term 1 FOLLOWER]: Refusing update from remote peer e95f5e34e1b6473d82783d1f36a2326b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:10:55.725113 18282 raft_consensus.cc:1261] T ea9519dd5bf340ab9bb47ed576eb9589 P 058f531272aa484fbcd67affbeafce0c [term 1 FOLLOWER]: Refusing update from remote peer e95f5e34e1b6473d82783d1f36a2326b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:10:55.726122 18616 consensus_queue.cc:1035] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9a1d643d3004ec0b48ef3f8d8d11c00" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:10:55.726540 18612 consensus_queue.cc:1035] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b [LEADER]: Connected to new peer: Peer: permanent_uuid: "058f531272aa484fbcd67affbeafce0c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 03:10:55.731945 18459 tablet.cc:2369] T ea9519dd5bf340ab9bb47ed576eb9589 P d9a1d643d3004ec0b48ef3f8d8d11c00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:10:55.752939 18140 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:10:55.753315 18140 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:10:55.761004 18140 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 787695866aa742f69507849b3e7a09c8: Sending DeleteTablet for 3 replicas of tablet ea9519dd5bf340ab9bb47ed576eb9589
I20260926 03:10:55.761613 18262 tablet_service.cc:1514] Processing DeleteTablet for tablet ea9519dd5bf340ab9bb47ed576eb9589 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:10:55 UTC) from {username='slave'} at 127.0.0.1:42162
I20260926 03:10:55.762143 18523 tablet_service.cc:1514] Processing DeleteTablet for tablet ea9519dd5bf340ab9bb47ed576eb9589 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:10:55 UTC) from {username='slave'} at 127.0.0.1:43060
I20260926 03:10:55.763056 18393 tablet_service.cc:1514] Processing DeleteTablet for tablet ea9519dd5bf340ab9bb47ed576eb9589 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:10:55 UTC) from {username='slave'} at 127.0.0.1:60570
I20260926 03:10:55.763314 18261 tablet_service.cc:1514] Processing DeleteTablet for tablet ea9519dd5bf340ab9bb47ed576eb9589 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:10:55 UTC) from {username='slave'} at 127.0.0.1:42162
I20260926 03:10:55.763752 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 18200
I20260926 03:10:55.770045 18630 tablet_replica.cc:330] T ea9519dd5bf340ab9bb47ed576eb9589 P d9a1d643d3004ec0b48ef3f8d8d11c00: stopping tablet replica
I20260926 03:10:55.773655 18628 tablet_replica.cc:330] T ea9519dd5bf340ab9bb47ed576eb9589 P 058f531272aa484fbcd67affbeafce0c: stopping tablet replica
I20260926 03:10:55.774183 18628 raft_consensus.cc:2229] T ea9519dd5bf340ab9bb47ed576eb9589 P 058f531272aa484fbcd67affbeafce0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:10:55.774717 18628 raft_consensus.cc:2258] T ea9519dd5bf340ab9bb47ed576eb9589 P 058f531272aa484fbcd67affbeafce0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:10:55.778827 18630 raft_consensus.cc:2229] T ea9519dd5bf340ab9bb47ed576eb9589 P d9a1d643d3004ec0b48ef3f8d8d11c00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:10:55.779897 18630 raft_consensus.cc:2258] T ea9519dd5bf340ab9bb47ed576eb9589 P d9a1d643d3004ec0b48ef3f8d8d11c00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:10:55.780857 18629 tablet_replica.cc:330] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b: stopping tablet replica
I20260926 03:10:55.781459 18629 raft_consensus.cc:2229] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:10:55.782218 18629 raft_consensus.cc:2258] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:10:55.782888 18630 ts_tablet_manager.cc:1905] T ea9519dd5bf340ab9bb47ed576eb9589 P d9a1d643d3004ec0b48ef3f8d8d11c00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:10:55.783191 18628 ts_tablet_manager.cc:1905] T ea9519dd5bf340ab9bb47ed576eb9589 P 058f531272aa484fbcd67affbeafce0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:10:55.790238 18601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:55.766919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42174 (local address 127.0.97.1:34955)
0926 03:10:55.767096 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:55.767104 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:55.778808 (+ 11704us) server_negotiation.cc:407] Connection header received
0926 03:10:55.779859 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:55.779868 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:55.780037 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:55.780127 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:55.780900 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.782053 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.786095 (+ 4042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.786770 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.786875 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:55.788423 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:55.788435 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:55.788454 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:10:55.789314 (+ 860us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:55.789384 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:10:55.789410 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:55.789497 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:55.789739 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 03:10:55.789861 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 03:10:55.790205 18630 ts_tablet_manager.cc:1918] T ea9519dd5bf340ab9bb47ed576eb9589 P d9a1d643d3004ec0b48ef3f8d8d11c00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:10:55.790393 18630 log.cc:1196] T ea9519dd5bf340ab9bb47ed576eb9589 P d9a1d643d3004ec0b48ef3f8d8d11c00: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-1/wal/wals/ea9519dd5bf340ab9bb47ed576eb9589
I20260926 03:10:55.790977 18630 ts_tablet_manager.cc:1939] T ea9519dd5bf340ab9bb47ed576eb9589 P d9a1d643d3004ec0b48ef3f8d8d11c00: Deleting consensus metadata
I20260926 03:10:55.791580 18322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:10:55.792954 18629 ts_tablet_manager.cc:1905] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:10:55.793001 18127 catalog_manager.cc:4841] TS d9a1d643d3004ec0b48ef3f8d8d11c00 (127.0.97.2:34811): tablet ea9519dd5bf340ab9bb47ed576eb9589 (table test-table [id=f20c71ae25db46d49f4f89030bc362eb]) successfully deleted
I20260926 03:10:55.790218 18628 ts_tablet_manager.cc:1918] T ea9519dd5bf340ab9bb47ed576eb9589 P 058f531272aa484fbcd67affbeafce0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:10:55.794152 18628 log.cc:1196] T ea9519dd5bf340ab9bb47ed576eb9589 P 058f531272aa484fbcd67affbeafce0c: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-0/wal/wals/ea9519dd5bf340ab9bb47ed576eb9589
I20260926 03:10:55.800701 18629 ts_tablet_manager.cc:1918] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:10:55.800894 18629 log.cc:1196] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391950058021-388-0/minicluster-data/ts-2/wal/wals/ea9519dd5bf340ab9bb47ed576eb9589
I20260926 03:10:55.801393 18629 ts_tablet_manager.cc:1939] T ea9519dd5bf340ab9bb47ed576eb9589 P e95f5e34e1b6473d82783d1f36a2326b: Deleting consensus metadata
I20260926 03:10:55.803440 18126 catalog_manager.cc:4841] TS e95f5e34e1b6473d82783d1f36a2326b (127.0.97.3:35009): tablet ea9519dd5bf340ab9bb47ed576eb9589 (table test-table [id=f20c71ae25db46d49f4f89030bc362eb]) successfully deleted
I20260926 03:10:55.915613 18628 ts_tablet_manager.cc:1939] T ea9519dd5bf340ab9bb47ed576eb9589 P 058f531272aa484fbcd67affbeafce0c: Deleting consensus metadata
I20260926 03:10:55.915740 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 18200
W20260926 03:10:55.931460 18127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:34955 (error 108)
I20260926 03:10:55.932201 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 18331
I20260926 03:10:55.933655 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:55.933004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34955 (local address 127.0.0.1:42190)
0926 03:10:55.933332 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:55.933460 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
W20260926 03:10:55.933949 18127 catalog_manager.cc:4584] TS 058f531272aa484fbcd67affbeafce0c (127.0.97.1:34955): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea9519dd5bf340ab9bb47ed576eb9589: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34955: connect: Connection refused (error 111)
I20260926 03:10:55.939543 18605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:55.933315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60574 (local address 127.0.97.2:34811)
0926 03:10:55.933473 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:55.933478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:55.933493 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:10:55.933546 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:55.933549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:55.933638 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:55.933727 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:55.934165 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.935005 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.936308 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:55.936698 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:55.936814 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:55.937939 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:55.937948 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:55.937962 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:10:55.938781 (+ 819us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:55.938840 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:10:55.938861 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:55.938972 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:55.939102 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 03:10:55.939216 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:10:55.940186 18453 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:10:55.989470 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:55.988893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34955 (local address 127.0.0.1:42196)
0926 03:10:55.989171 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:55.989268 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260926 03:10:55.990666 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:55.990183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34955 (local address 127.0.0.1:42210)
0926 03:10:55.990387 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:55.990465 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260926 03:10:56.035142 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 18331
I20260926 03:10:56.050221 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 18461
I20260926 03:10:56.051959 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:56.051470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34955 (local address 127.0.0.1:42214)
0926 03:10:56.051676 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:56.051780 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 03:10:56.053161 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:56.052756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34955 (local address 127.0.0.1:42226)
0926 03:10:56.052925 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:56.053013 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 03:10:56.058956 18603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:56.051023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43082 (local address 127.0.97.3:35009)
0926 03:10:56.051185 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:56.051190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:56.051207 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:10:56.051438 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:56.051444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:56.051572 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:56.051676 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:56.052265 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:56.053320 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:56.055493 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:56.055996 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:56.056110 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:56.057329 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:56.057337 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:56.057353 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:10:56.057987 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:56.058037 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:10:56.058059 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:56.058146 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:56.058432 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 03:10:56.058563 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 03:10:56.059649 18583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:10:56.133725 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:56.133203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34955 (local address 127.0.0.1:42238)
0926 03:10:56.133459 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:56.133564 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260926 03:10:56.134994 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:56.134449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34955 (local address 127.0.0.1:42248)
0926 03:10:56.134720 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:56.134828 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260926 03:10:56.191001 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 18461
I20260926 03:10:56.207175 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 18110
I20260926 03:10:56.208515 18171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:10:56.317936 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:56.317304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34955 (local address 127.0.0.1:42254)
0926 03:10:56.317530 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:56.317670 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 03:10:56.318027 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 18110
2026-09-26T03:10:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2847 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:10:56 dist-test-slave-hmbw krb5kdc[18639](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:10:56 dist-test-slave-hmbw krb5kdc[18639](info): set up 2 sockets
Sep 26 03:10:56 dist-test-slave-hmbw krb5kdc[18639](info): commencing operation
krb5kdc: starting...
W20260926 03:10:57.346663 388 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:10:57 dist-test-slave-hmbw krb5kdc[18639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:10:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:10:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:10:57.607451 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43959
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:10:57.946110 18652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:10:57.946448 18652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:10:57.946604 18652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:10:57.954037 18652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:10:57.954182 18652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:10:57.954249 18652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:10:57.954362 18652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:10:57.954427 18652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:10:57.967288 18652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:10:57.969249 18652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:10:57.970774 18652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:10:57.983896 18657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:57.983798 18658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:57.983814 18660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:10:57.984570 18652 server_base.cc:1034] running on GCE node
I20260926 03:10:57.985505 18652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:10:57.987169 18652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:10:57.988447 18652 hybrid_clock.cc:648] HybridClock initialized: now 1790392257988412 us; error 91 us; skew 500 ppm
I20260926 03:10:57.988839 18652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:10:57 dist-test-slave-hmbw krb5kdc[18639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:10:57.992198 18652 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:10:57.994192 18652 webserver.cc:460] Webserver started at http://127.0.97.62:39065/ using document root <none> and password file <none>
I20260926 03:10:57.995246 18652 fs_manager.cc:359] Metadata directory not provided
I20260926 03:10:57.995422 18652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:10:57.995848 18652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:10:57.999082 18652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "720ba0498b594cd980c029997a474ea0"
format_stamp: "Formatted at 2026-09-26 03:10:57 on dist-test-slave-hmbw"
server_key: "1a18c72f3df55dacde04b2bcae810884"
server_key_iv: "5b06f878fc865459c0820c537df61892"
server_key_version: "encryptionkey@0"
I20260926 03:10:57.999883 18652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "720ba0498b594cd980c029997a474ea0"
format_stamp: "Formatted at 2026-09-26 03:10:57 on dist-test-slave-hmbw"
server_key: "1a18c72f3df55dacde04b2bcae810884"
server_key_iv: "5b06f878fc865459c0820c537df61892"
server_key_version: "encryptionkey@0"
I20260926 03:10:58.005756 18652 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 03:10:58.010021 18667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:58.011677 18652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:10:58.011861 18652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "720ba0498b594cd980c029997a474ea0"
format_stamp: "Formatted at 2026-09-26 03:10:57 on dist-test-slave-hmbw"
server_key: "1a18c72f3df55dacde04b2bcae810884"
server_key_iv: "5b06f878fc865459c0820c537df61892"
server_key_version: "encryptionkey@0"
I20260926 03:10:58.012073 18652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:10:58.058983 18652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:10:58.063488 18652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:10:58.063755 18652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:10:58.081825 18652 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43959
I20260926 03:10:58.081843 18718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43959 every 8 connection(s)
I20260926 03:10:58.083384 18652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:10:58.087587 18719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:10:58.092562 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 18652
I20260926 03:10:58.092749 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:10:58.093212 388 external_mini_cluster.cc:1428] Setting key 3032ed0517df7786f42e989684ab22ae
I20260926 03:10:58.100373 18719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 720ba0498b594cd980c029997a474ea0: Bootstrap starting.
I20260926 03:10:58.104260 18719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 720ba0498b594cd980c029997a474ea0: Neither blocks nor log segments found. Creating new log.
I20260926 03:10:58.105537 18719 log.cc:824] T 00000000000000000000000000000000 P 720ba0498b594cd980c029997a474ea0: Log is configured to *not* fsync() on all Append() calls
I20260926 03:10:58.109122 18719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 720ba0498b594cd980c029997a474ea0: No bootstrap required, opened a new log
Sep 26 03:10:58 dist-test-slave-hmbw krb5kdc[18639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:10:58.119813 18719 raft_consensus.cc:348] T 00000000000000000000000000000000 P 720ba0498b594cd980c029997a474ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "720ba0498b594cd980c029997a474ea0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43959 } }
I20260926 03:10:58.120287 18722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:58.095178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54020 (local address 127.0.97.62:43959)
0926 03:10:58.096021 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:58.096038 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:58.096096 (+ 58us) server_negotiation.cc:407] Connection header received
0926 03:10:58.098639 (+ 2543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:58.098673 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:58.099230 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:58.099717 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:58.101234 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:58.103264 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:58.104797 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:58.105860 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:58.106073 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:58.110534 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:58.110547 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:58.110571 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:10:58.110630 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:58.114890 (+ 4260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:58.115884 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:58.115895 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:58.115906 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:58.116159 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:58.116647 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:58.116652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:58.116656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:58.117518 (+ 862us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:58.117852 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:58.118321 (+ 469us) server_negotiation.cc:299] Negotiation successful
0926 03:10:58.118882 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":352,"threads_started":1}
I20260926 03:10:58.120322 18719 raft_consensus.cc:374] T 00000000000000000000000000000000 P 720ba0498b594cd980c029997a474ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:10:58.120416 18719 raft_consensus.cc:729] T 00000000000000000000000000000000 P 720ba0498b594cd980c029997a474ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 720ba0498b594cd980c029997a474ea0, State: Initialized, Role: FOLLOWER
I20260926 03:10:58.121044 18719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 720ba0498b594cd980c029997a474ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "720ba0498b594cd980c029997a474ea0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43959 } }
I20260926 03:10:58.121289 18719 raft_consensus.cc:388] T 00000000000000000000000000000000 P 720ba0498b594cd980c029997a474ea0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:10:58.121389 18719 raft_consensus.cc:482] T 00000000000000000000000000000000 P 720ba0498b594cd980c029997a474ea0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:10:58.121520 18719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 720ba0498b594cd980c029997a474ea0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:10:58.124470 18719 raft_consensus.cc:504] T 00000000000000000000000000000000 P 720ba0498b594cd980c029997a474ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "720ba0498b594cd980c029997a474ea0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43959 } }
I20260926 03:10:58.124985 18719 leader_election.cc:302] T 00000000000000000000000000000000 P 720ba0498b594cd980c029997a474ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 720ba0498b594cd980c029997a474ea0; no voters:
I20260926 03:10:58.125576 18719 leader_election.cc:288] T 00000000000000000000000000000000 P 720ba0498b594cd980c029997a474ea0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:10:58.125758 18724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 720ba0498b594cd980c029997a474ea0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:10:58.126767 18724 raft_consensus.cc:686] T 00000000000000000000000000000000 P 720ba0498b594cd980c029997a474ea0 [term 1 LEADER]: Becoming Leader. State: Replica: 720ba0498b594cd980c029997a474ea0, State: Running, Role: LEADER
I20260926 03:10:58.127369 18724 consensus_queue.cc:237] T 00000000000000000000000000000000 P 720ba0498b594cd980c029997a474ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "720ba0498b594cd980c029997a474ea0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43959 } }
I20260926 03:10:58.127897 18719 sys_catalog.cc:564] T 00000000000000000000000000000000 P 720ba0498b594cd980c029997a474ea0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:10:58.130442 18726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 720ba0498b594cd980c029997a474ea0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 720ba0498b594cd980c029997a474ea0. Latest consensus state: current_term: 1 leader_uuid: "720ba0498b594cd980c029997a474ea0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "720ba0498b594cd980c029997a474ea0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43959 } } }
I20260926 03:10:58.130946 18726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 720ba0498b594cd980c029997a474ea0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:10:58.130478 18725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 720ba0498b594cd980c029997a474ea0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "720ba0498b594cd980c029997a474ea0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "720ba0498b594cd980c029997a474ea0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43959 } } }
I20260926 03:10:58.131578 18725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 720ba0498b594cd980c029997a474ea0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:10:58.133074 18732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:10:58.138453 18732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:10:58.148511 18732 catalog_manager.cc:1347] Generated new cluster ID: f3096905bb284617b692a4833afdbbbb
I20260926 03:10:58.148645 18732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:10:58.159984 18732 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:10:58.161468 18732 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:10:58.172796 18732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 720ba0498b594cd980c029997a474ea0: Generated new TSK 0
I20260926 03:10:58.174058 18732 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:10:58.242792 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43959
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:10:58.575896 18745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:10:58.576200 18745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:10:58.576333 18745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:10:58.583714 18745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:10:58.583845 18745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:10:58.583992 18745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:10:58.594086 18745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:10:58.595438 18745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:10:58.596696 18745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:10:58.609771 18750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:58.609741 18751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:58.609858 18753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:10:58.610841 18745 server_base.cc:1034] running on GCE node
I20260926 03:10:58.611459 18745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:10:58.612512 18745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:10:58.613767 18745 hybrid_clock.cc:648] HybridClock initialized: now 1790392258613706 us; error 60 us; skew 500 ppm
I20260926 03:10:58.614108 18745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:10:58 dist-test-slave-hmbw krb5kdc[18639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:10:58.617671 18745 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:10:58.619284 18745 webserver.cc:460] Webserver started at http://127.0.97.1:36719/ using document root <none> and password file <none>
I20260926 03:10:58.619979 18745 fs_manager.cc:359] Metadata directory not provided
I20260926 03:10:58.620086 18745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:10:58.620335 18745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:10:58.623569 18745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-0/data/instance:
uuid: "cf24ac44026946698ac2f93cb5f895d8"
format_stamp: "Formatted at 2026-09-26 03:10:58 on dist-test-slave-hmbw"
server_key: "2a168518e103060c43959d25954f3983"
server_key_iv: "a8d8ac684123ea6236533507f45e78b5"
server_key_version: "encryptionkey@0"
I20260926 03:10:58.624850 18745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-0/wal/instance:
uuid: "cf24ac44026946698ac2f93cb5f895d8"
format_stamp: "Formatted at 2026-09-26 03:10:58 on dist-test-slave-hmbw"
server_key: "2a168518e103060c43959d25954f3983"
server_key_iv: "a8d8ac684123ea6236533507f45e78b5"
server_key_version: "encryptionkey@0"
I20260926 03:10:58.630962 18745 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 03:10:58.634970 18760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:58.636703 18745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:10:58.636894 18745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "cf24ac44026946698ac2f93cb5f895d8"
format_stamp: "Formatted at 2026-09-26 03:10:58 on dist-test-slave-hmbw"
server_key: "2a168518e103060c43959d25954f3983"
server_key_iv: "a8d8ac684123ea6236533507f45e78b5"
server_key_version: "encryptionkey@0"
I20260926 03:10:58.637091 18745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:10:58.658212 18745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:10:58.663580 18745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:10:58.663858 18745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:10:58.664860 18745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:10:58.666330 18745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:10:58.666409 18745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:58.666523 18745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:10:58.666569 18745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260926 03:10:58.697881 18745 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44263
I20260926 03:10:58.697952 18872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44263 every 8 connection(s)
I20260926 03:10:58.699218 18745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:10:58.699571 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 18745
I20260926 03:10:58.699753 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:10:58.700299 388 external_mini_cluster.cc:1428] Setting key 003caf32cb292c2669bfb70fbf6513a9
Sep 26 03:10:58 dist-test-slave-hmbw krb5kdc[18639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:10:58.721741 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:58.702073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45107 (local address 127.0.97.62:43959)
0926 03:10:58.702621 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:58.702627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:58.703864 (+ 1237us) server_negotiation.cc:407] Connection header received
0926 03:10:58.705732 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:58.705739 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:58.705843 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:58.705940 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:58.707617 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:58.708346 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:58.710170 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:58.710680 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:58.710797 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:58.715131 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:58.715137 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:58.715140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:10:58.715179 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:58.717808 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:58.718776 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:58.718780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:58.718783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:58.718931 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:58.719431 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:58.719435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:58.719437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:58.719856 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:58.720039 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:58.721046 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0926 03:10:58.721347 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":268,"threads_started":1}
I20260926 03:10:58.722862 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:58.702773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43959 (local address 127.0.97.1:45107)
0926 03:10:58.703684 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:58.703724 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:58.705442 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:58.706202 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:58.706212 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:58.706798 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:10:58.707428 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:58.707448 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:58.708508 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:58.708512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:58.709983 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:58.710000 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:58.710923 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:58.710929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:58.711237 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:58.712001 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:10:58.712036 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:10:58.714851 (+ 2815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:10:58.718017 (+ 3166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:10:58.718028 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:10:58.718047 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:10:58.718619 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:10:58.719058 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:10:58.719064 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:10:58.719068 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:10:58.719291 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:10:58.720033 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:10:58.720043 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:10:58.720463 (+ 420us) client_negotiation.cc:769] Sending connection context
0926 03:10:58.720887 (+ 424us) client_negotiation.cc:240] Negotiation successful
0926 03:10:58.721529 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":358,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:10:58.724689 18873 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43959
I20260926 03:10:58.725069 18873 heartbeater.cc:464] Registering TS with master...
I20260926 03:10:58.725868 18873 heartbeater.cc:511] Master 127.0.97.62:43959 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 03:10:58.728364 18684 ts_manager.cc:194] Registered new tserver with Master: cf24ac44026946698ac2f93cb5f895d8 (127.0.97.1:44263)
I20260926 03:10:58.729961 18684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:10:58.758286 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43959
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:10:59.085908 18879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:10:59.086203 18879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:10:59.086323 18879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:10:59.093899 18879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:10:59.094033 18879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:10:59.094185 18879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:10:59.104375 18879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:10:59.105700 18879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:10:59.107062 18879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:10:59.120471 18885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:59.120523 18884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:59.120558 18887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:10:59.121377 18879 server_base.cc:1034] running on GCE node
I20260926 03:10:59.122073 18879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:10:59.123112 18879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:10:59.124328 18879 hybrid_clock.cc:648] HybridClock initialized: now 1790392259124270 us; error 51 us; skew 500 ppm
I20260926 03:10:59.124693 18879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:10:59 dist-test-slave-hmbw krb5kdc[18639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:10:59.136590 18879 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:10:59.138293 18879 webserver.cc:460] Webserver started at http://127.0.97.2:34359/ using document root <none> and password file <none>
I20260926 03:10:59.139077 18879 fs_manager.cc:359] Metadata directory not provided
I20260926 03:10:59.139204 18879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:10:59.139506 18879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:10:59.142730 18879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-1/data/instance:
uuid: "a1adcafff96246f8af5aa7f27ef2fd9a"
format_stamp: "Formatted at 2026-09-26 03:10:59 on dist-test-slave-hmbw"
server_key: "f0f5bb1dabf4e34e096b074d8f2b7204"
server_key_iv: "7d282780eef34a1bc0a660cf81cf544f"
server_key_version: "encryptionkey@0"
I20260926 03:10:59.143549 18879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a1adcafff96246f8af5aa7f27ef2fd9a"
format_stamp: "Formatted at 2026-09-26 03:10:59 on dist-test-slave-hmbw"
server_key: "f0f5bb1dabf4e34e096b074d8f2b7204"
server_key_iv: "7d282780eef34a1bc0a660cf81cf544f"
server_key_version: "encryptionkey@0"
I20260926 03:10:59.149181 18879 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 03:10:59.152984 18894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:59.154417 18879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 03:10:59.154609 18879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "a1adcafff96246f8af5aa7f27ef2fd9a"
format_stamp: "Formatted at 2026-09-26 03:10:59 on dist-test-slave-hmbw"
server_key: "f0f5bb1dabf4e34e096b074d8f2b7204"
server_key_iv: "7d282780eef34a1bc0a660cf81cf544f"
server_key_version: "encryptionkey@0"
I20260926 03:10:59.154855 18879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:10:59.171607 18879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:10:59.176476 18879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:10:59.176743 18879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:10:59.177719 18879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:10:59.179133 18879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:10:59.179216 18879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:59.179334 18879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:10:59.179390 18879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:59.210296 18879 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33163
I20260926 03:10:59.210342 19006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33163 every 8 connection(s)
I20260926 03:10:59.211651 18879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:10:59.212056 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 18879
I20260926 03:10:59.212212 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:10:59.212729 388 external_mini_cluster.cc:1428] Setting key dadf913781dec96423412d67a501582e
Sep 26 03:10:59 dist-test-slave-hmbw krb5kdc[18639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:10:59.233579 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:59.214311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37925 (local address 127.0.97.62:43959)
0926 03:10:59.214564 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:59.214569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:59.216095 (+ 1526us) server_negotiation.cc:407] Connection header received
0926 03:10:59.218088 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:59.218093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:59.218182 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:59.218267 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:59.219882 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.220593 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.222205 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.222685 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.222804 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:59.226702 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:59.226709 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:59.226714 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:10:59.226759 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:59.230012 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:59.230939 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:59.230944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:59.230948 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:59.231122 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:59.231600 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:59.231604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:59.231608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:59.231964 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:59.232105 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:59.232965 (+ 860us) server_negotiation.cc:299] Negotiation successful
0926 03:10:59.233214 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260926 03:10:59.234586 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:59.215006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43959 (local address 127.0.97.2:37925)
0926 03:10:59.215888 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:59.215979 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:59.217790 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:59.218564 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:59.218574 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:59.219107 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:10:59.219702 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:59.219719 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.220763 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.220767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:59.222050 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:59.222062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.222931 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.222939 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:59.223164 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:59.223844 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:10:59.223878 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:10:59.226330 (+ 2452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:10:59.230199 (+ 3869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:10:59.230207 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:10:59.230224 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:10:59.230800 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:10:59.231220 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:10:59.231223 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:10:59.231225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:10:59.231487 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:10:59.232082 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:10:59.232088 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:10:59.232511 (+ 423us) client_negotiation.cc:769] Sending connection context
0926 03:10:59.232750 (+ 239us) client_negotiation.cc:240] Negotiation successful
0926 03:10:59.233297 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":344,"threads_started":1}
I20260926 03:10:59.235983 19007 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43959
I20260926 03:10:59.236253 19007 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 03:10:59.237022 19007 heartbeater.cc:511] Master 127.0.97.62:43959 requested a full tablet report, sending...
I20260926 03:10:59.238642 18684 ts_manager.cc:194] Registered new tserver with Master: a1adcafff96246f8af5aa7f27ef2fd9a (127.0.97.2:33163)
I20260926 03:10:59.239596 18684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:10:59.270655 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43959
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:10:59.599231 19012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:10:59.599535 19012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:10:59.599629 19012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:10:59.607023 19012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:10:59.607136 19012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:10:59.607259 19012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:10:59.617455 19012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:10:59.618826 19012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:10:59.620129 19012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:10:59.633420 19017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:59.633483 19018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:10:59.633800 19020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:10:59.634989 19012 server_base.cc:1034] running on GCE node
I20260926 03:10:59.635602 19012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:10:59.636627 19012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:10:59.637850 19012 hybrid_clock.cc:648] HybridClock initialized: now 1790392259637808 us; error 36 us; skew 500 ppm
I20260926 03:10:59.638175 19012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:10:59 dist-test-slave-hmbw krb5kdc[18639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:10:59.641341 19012 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:10:59.642982 19012 webserver.cc:460] Webserver started at http://127.0.97.3:43037/ using document root <none> and password file <none>
I20260926 03:10:59.643724 19012 fs_manager.cc:359] Metadata directory not provided
I20260926 03:10:59.643862 19012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:10:59.644191 19012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:10:59.647212 19012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-2/data/instance:
uuid: "c36b7f5787e94a7494b5fabc9d73a233"
format_stamp: "Formatted at 2026-09-26 03:10:59 on dist-test-slave-hmbw"
server_key: "c1624d329e19eb9a35ff7e210b9dafe1"
server_key_iv: "11953e1d74104527feff13da7c145a18"
server_key_version: "encryptionkey@0"
I20260926 03:10:59.648021 19012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c36b7f5787e94a7494b5fabc9d73a233"
format_stamp: "Formatted at 2026-09-26 03:10:59 on dist-test-slave-hmbw"
server_key: "c1624d329e19eb9a35ff7e210b9dafe1"
server_key_iv: "11953e1d74104527feff13da7c145a18"
server_key_version: "encryptionkey@0"
I20260926 03:10:59.653448 19012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:10:59.657382 19027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:59.659024 19012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:10:59.659193 19012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "c36b7f5787e94a7494b5fabc9d73a233"
format_stamp: "Formatted at 2026-09-26 03:10:59 on dist-test-slave-hmbw"
server_key: "c1624d329e19eb9a35ff7e210b9dafe1"
server_key_iv: "11953e1d74104527feff13da7c145a18"
server_key_version: "encryptionkey@0"
I20260926 03:10:59.659485 19012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:10:59.681304 19012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:10:59.685953 19012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:10:59.686251 19012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:10:59.687386 19012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:10:59.688850 19012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:10:59.688948 19012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:59.689065 19012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:10:59.689117 19012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:10:59.721171 19012 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39123
I20260926 03:10:59.721215 19139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39123 every 8 connection(s)
I20260926 03:10:59.722422 19012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:10:59.724709 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 19012
I20260926 03:10:59.724895 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:10:59.725447 388 external_mini_cluster.cc:1428] Setting key eb486718b433c1b01fd5540b21b785cb
I20260926 03:10:59.733645 18873 heartbeater.cc:503] Master 127.0.97.62:43959 was elected leader, sending a full tablet report...
Sep 26 03:10:59 dist-test-slave-hmbw krb5kdc[18639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:10:59.743645 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:59.725113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40465 (local address 127.0.97.62:43959)
0926 03:10:59.725306 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:59.725311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:59.726809 (+ 1498us) server_negotiation.cc:407] Connection header received
0926 03:10:59.729041 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:59.729047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:59.729129 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:59.729209 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:59.730726 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.731499 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.733076 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.733497 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.733605 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:59.737250 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:59.737256 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:59.737259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:10:59.737309 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:59.740278 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:59.741402 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:59.741408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:59.741412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:59.741587 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:59.742025 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:59.742028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:59.742030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:59.742353 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:59.742472 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:59.743223 (+ 751us) server_negotiation.cc:299] Negotiation successful
0926 03:10:59.743403 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:10:59.744606 19142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:59.725696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43959 (local address 127.0.97.3:40465)
0926 03:10:59.726637 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:59.726701 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:59.728791 (+ 2090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:59.729460 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:59.729468 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:59.729965 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:10:59.730563 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:59.730584 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.731645 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.731649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:59.732933 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:59.732945 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.733713 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.733721 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:59.733961 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:59.734747 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:10:59.734772 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:10:59.737032 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:10:59.740491 (+ 3459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:10:59.740501 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:10:59.740521 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:10:59.741161 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:10:59.741697 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:10:59.741702 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:10:59.741706 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:10:59.741927 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:10:59.742477 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:10:59.742485 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:10:59.742853 (+ 368us) client_negotiation.cc:769] Sending connection context
0926 03:10:59.743116 (+ 263us) client_negotiation.cc:240] Negotiation successful
0926 03:10:59.743514 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":338,"threads_started":1}
I20260926 03:10:59.745978 19140 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43959
I20260926 03:10:59.746263 19140 heartbeater.cc:464] Registering TS with master...
I20260926 03:10:59.747010 19140 heartbeater.cc:511] Master 127.0.97.62:43959 requested a full tablet report, sending...
I20260926 03:10:59.748494 18684 ts_manager.cc:194] Registered new tserver with Master: c36b7f5787e94a7494b5fabc9d73a233 (127.0.97.3:39123)
I20260926 03:10:59.749300 18684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40465
I20260926 03:10:59.753963 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:10:59.772164 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:59.758140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.0.97.62:43959)
0926 03:10:59.758310 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:59.758315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:59.758826 (+ 511us) server_negotiation.cc:407] Connection header received
0926 03:10:59.758992 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:59.758996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:59.759065 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:59.759169 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:10:59.760757 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.761572 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.763097 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.763562 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.763664 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:59.766406 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:10:59.766412 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:10:59.766415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:10:59.766454 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:10:59.769064 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:59.769968 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:59.769972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:59.769979 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:59.770113 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:10:59.770678 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:10:59.770682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:10:59.770684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:10:59.771012 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:10:59.771245 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:59.771757 (+ 512us) server_negotiation.cc:299] Negotiation successful
0926 03:10:59.771915 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:10:59.776467 18684 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:10:59.783401 18684 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:10:59.806038 19152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:59.795377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39123 (local address 127.0.0.1:44668)
0926 03:10:59.796155 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:59.796198 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:59.796409 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:59.798265 (+ 1856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:59.798270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:59.798290 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:10:59.798820 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:59.798835 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.801195 (+ 2360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.801200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:59.803575 (+ 2375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:59.803588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.805054 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.805058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:59.805251 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:59.805307 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 03:10:59.805446 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 03:10:59.805604 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":161,"thread_start_us":319,"threads_started":1}
I20260926 03:10:59.806973 19154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:59.795666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44668 (local address 127.0.97.3:39123)
0926 03:10:59.797727 (+ 2061us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:59.797735 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:59.797764 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:10:59.797850 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:59.797858 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:59.798064 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:59.799084 (+ 1020us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:10:59.799376 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.801051 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.803739 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.804849 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.804996 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:59.805269 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:59.806384 (+ 1115us) server_negotiation.cc:299] Negotiation successful
0926 03:10:59.806599 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1894,"thread_start_us":1749,"threads_started":1}
I20260926 03:10:59.808367 19155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:59.796231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44210 (local address 127.0.97.2:33163)
0926 03:10:59.796784 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:59.796793 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:59.797217 (+ 424us) server_negotiation.cc:407] Connection header received
0926 03:10:59.797332 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:59.797341 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:59.797599 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:59.797789 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:10:59.799434 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.801027 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.804551 (+ 3524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.805924 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.806053 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:59.806230 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:59.807038 (+ 808us) server_negotiation.cc:299] Negotiation successful
0926 03:10:59.808044 (+ 1006us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":247,"threads_started":1}
I20260926 03:10:59.809155 19153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:59.795729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33163 (local address 127.0.0.1:44210)
0926 03:10:59.796951 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:59.796969 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:59.797090 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:59.798064 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:59.798071 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:59.798100 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:10:59.799224 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:59.799237 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.801195 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.801200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:59.804363 (+ 3163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:59.804380 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.806628 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.806634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:59.806798 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:59.806830 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 03:10:59.808617 (+ 1787us) client_negotiation.cc:240] Negotiation successful
0926 03:10:59.808765 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1078,"mutex_wait_us":884,"thread_start_us":960,"threads_started":1}
I20260926 03:10:59.809376 19157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:59.797989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45426 (local address 127.0.97.1:44263)
0926 03:10:59.799611 (+ 1622us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:59.799617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:59.799639 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:10:59.799710 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:59.799716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:59.799867 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:59.800029 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:10:59.800560 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.801881 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.806336 (+ 4455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.807615 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.807767 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:59.808003 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:59.808889 (+ 886us) server_negotiation.cc:299] Negotiation successful
0926 03:10:59.809074 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1486,"thread_start_us":1322,"threads_started":1}
I20260926 03:10:59.812994 18942 tablet_service.cc:1467] Processing CreateTablet for tablet dd3578d5d53647678b20435a1e4c5516 (DEFAULT_TABLE table=test-table [id=f3047413908f442a90059b6419d46856]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:10:59.812592 19075 tablet_service.cc:1467] Processing CreateTablet for tablet dd3578d5d53647678b20435a1e4c5516 (DEFAULT_TABLE table=test-table [id=f3047413908f442a90059b6419d46856]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:10:59.814412 18942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd3578d5d53647678b20435a1e4c5516. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:10:59.814409 19075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd3578d5d53647678b20435a1e4c5516. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:10:59.821221 19156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:59.797670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44263 (local address 127.0.0.1:45426)
0926 03:10:59.798804 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:59.798822 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:59.798936 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:59.800048 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:59.800052 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:59.800075 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:10:59.800393 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:59.800403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.802076 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.802081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:59.804379 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:59.804389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.808140 (+ 3751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.808146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:59.808324 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:59.808360 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 03:10:59.808450 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 03:10:59.808595 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":990,"mutex_wait_us":37,"thread_start_us":820,"threads_started":1}
I20260926 03:10:59.825815 18808 tablet_service.cc:1467] Processing CreateTablet for tablet dd3578d5d53647678b20435a1e4c5516 (DEFAULT_TABLE table=test-table [id=f3047413908f442a90059b6419d46856]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:10:59.827432 18808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd3578d5d53647678b20435a1e4c5516. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:10:59.829486 19158 tablet_bootstrap.cc:492] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a: Bootstrap starting.
I20260926 03:10:59.832566 19158 tablet_bootstrap.cc:654] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a: Neither blocks nor log segments found. Creating new log.
I20260926 03:10:59.833632 19158 log.cc:824] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a: Log is configured to *not* fsync() on all Append() calls
I20260926 03:10:59.836385 19158 tablet_bootstrap.cc:492] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a: No bootstrap required, opened a new log
I20260926 03:10:59.836731 19158 ts_tablet_manager.cc:1397] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 03:10:59.840519 19160 tablet_bootstrap.cc:492] T dd3578d5d53647678b20435a1e4c5516 P c36b7f5787e94a7494b5fabc9d73a233: Bootstrap starting.
I20260926 03:10:59.843811 19160 tablet_bootstrap.cc:654] T dd3578d5d53647678b20435a1e4c5516 P c36b7f5787e94a7494b5fabc9d73a233: Neither blocks nor log segments found. Creating new log.
I20260926 03:10:59.844822 19160 log.cc:824] T dd3578d5d53647678b20435a1e4c5516 P c36b7f5787e94a7494b5fabc9d73a233: Log is configured to *not* fsync() on all Append() calls
I20260926 03:10:59.846158 19161 tablet_bootstrap.cc:492] T dd3578d5d53647678b20435a1e4c5516 P cf24ac44026946698ac2f93cb5f895d8: Bootstrap starting.
I20260926 03:10:59.847229 19158 raft_consensus.cc:348] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36b7f5787e94a7494b5fabc9d73a233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39123 } } peers { permanent_uuid: "a1adcafff96246f8af5aa7f27ef2fd9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33163 } } peers { permanent_uuid: "cf24ac44026946698ac2f93cb5f895d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44263 } }
I20260926 03:10:59.847674 19158 raft_consensus.cc:374] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:10:59.847783 19158 raft_consensus.cc:729] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1adcafff96246f8af5aa7f27ef2fd9a, State: Initialized, Role: FOLLOWER
I20260926 03:10:59.848124 19160 tablet_bootstrap.cc:492] T dd3578d5d53647678b20435a1e4c5516 P c36b7f5787e94a7494b5fabc9d73a233: No bootstrap required, opened a new log
I20260926 03:10:59.848415 19160 ts_tablet_manager.cc:1397] T dd3578d5d53647678b20435a1e4c5516 P c36b7f5787e94a7494b5fabc9d73a233: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 03:10:59.848384 19158 consensus_queue.cc:260] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36b7f5787e94a7494b5fabc9d73a233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39123 } } peers { permanent_uuid: "a1adcafff96246f8af5aa7f27ef2fd9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33163 } } peers { permanent_uuid: "cf24ac44026946698ac2f93cb5f895d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44263 } }
I20260926 03:10:59.849552 19007 heartbeater.cc:503] Master 127.0.97.62:43959 was elected leader, sending a full tablet report...
I20260926 03:10:59.850015 19158 ts_tablet_manager.cc:1428] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260926 03:10:59.851948 19161 tablet_bootstrap.cc:654] T dd3578d5d53647678b20435a1e4c5516 P cf24ac44026946698ac2f93cb5f895d8: Neither blocks nor log segments found. Creating new log.
I20260926 03:10:59.853253 19161 log.cc:824] T dd3578d5d53647678b20435a1e4c5516 P cf24ac44026946698ac2f93cb5f895d8: Log is configured to *not* fsync() on all Append() calls
I20260926 03:10:59.857224 19160 raft_consensus.cc:348] T dd3578d5d53647678b20435a1e4c5516 P c36b7f5787e94a7494b5fabc9d73a233 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36b7f5787e94a7494b5fabc9d73a233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39123 } } peers { permanent_uuid: "a1adcafff96246f8af5aa7f27ef2fd9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33163 } } peers { permanent_uuid: "cf24ac44026946698ac2f93cb5f895d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44263 } }
I20260926 03:10:59.857533 19161 tablet_bootstrap.cc:492] T dd3578d5d53647678b20435a1e4c5516 P cf24ac44026946698ac2f93cb5f895d8: No bootstrap required, opened a new log
I20260926 03:10:59.857594 19160 raft_consensus.cc:374] T dd3578d5d53647678b20435a1e4c5516 P c36b7f5787e94a7494b5fabc9d73a233 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:10:59.857682 19160 raft_consensus.cc:729] T dd3578d5d53647678b20435a1e4c5516 P c36b7f5787e94a7494b5fabc9d73a233 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c36b7f5787e94a7494b5fabc9d73a233, State: Initialized, Role: FOLLOWER
I20260926 03:10:59.857816 19161 ts_tablet_manager.cc:1397] T dd3578d5d53647678b20435a1e4c5516 P cf24ac44026946698ac2f93cb5f895d8: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260926 03:10:59.858304 19160 consensus_queue.cc:260] T dd3578d5d53647678b20435a1e4c5516 P c36b7f5787e94a7494b5fabc9d73a233 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36b7f5787e94a7494b5fabc9d73a233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39123 } } peers { permanent_uuid: "a1adcafff96246f8af5aa7f27ef2fd9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33163 } } peers { permanent_uuid: "cf24ac44026946698ac2f93cb5f895d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44263 } }
I20260926 03:10:59.859515 19140 heartbeater.cc:503] Master 127.0.97.62:43959 was elected leader, sending a full tablet report...
I20260926 03:10:59.860111 19160 ts_tablet_manager.cc:1428] T dd3578d5d53647678b20435a1e4c5516 P c36b7f5787e94a7494b5fabc9d73a233: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260926 03:10:59.866353 19161 raft_consensus.cc:348] T dd3578d5d53647678b20435a1e4c5516 P cf24ac44026946698ac2f93cb5f895d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36b7f5787e94a7494b5fabc9d73a233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39123 } } peers { permanent_uuid: "a1adcafff96246f8af5aa7f27ef2fd9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33163 } } peers { permanent_uuid: "cf24ac44026946698ac2f93cb5f895d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44263 } }
I20260926 03:10:59.866822 19161 raft_consensus.cc:374] T dd3578d5d53647678b20435a1e4c5516 P cf24ac44026946698ac2f93cb5f895d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:10:59.866935 19161 raft_consensus.cc:729] T dd3578d5d53647678b20435a1e4c5516 P cf24ac44026946698ac2f93cb5f895d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf24ac44026946698ac2f93cb5f895d8, State: Initialized, Role: FOLLOWER
I20260926 03:10:59.867533 19161 consensus_queue.cc:260] T dd3578d5d53647678b20435a1e4c5516 P cf24ac44026946698ac2f93cb5f895d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36b7f5787e94a7494b5fabc9d73a233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39123 } } peers { permanent_uuid: "a1adcafff96246f8af5aa7f27ef2fd9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33163 } } peers { permanent_uuid: "cf24ac44026946698ac2f93cb5f895d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44263 } }
I20260926 03:10:59.868810 19163 raft_consensus.cc:482] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:10:59.869197 19161 ts_tablet_manager.cc:1428] T dd3578d5d53647678b20435a1e4c5516 P cf24ac44026946698ac2f93cb5f895d8: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260926 03:10:59.869057 19163 raft_consensus.cc:504] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36b7f5787e94a7494b5fabc9d73a233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39123 } } peers { permanent_uuid: "a1adcafff96246f8af5aa7f27ef2fd9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33163 } } peers { permanent_uuid: "cf24ac44026946698ac2f93cb5f895d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44263 } }
I20260926 03:10:59.870890 19163 leader_election.cc:288] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c36b7f5787e94a7494b5fabc9d73a233 (127.0.97.3:39123), cf24ac44026946698ac2f93cb5f895d8 (127.0.97.1:44263)
I20260926 03:10:59.879833 19157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:59.871578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48231 (local address 127.0.97.1:44263)
0926 03:10:59.871756 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:59.871762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:59.872696 (+ 934us) server_negotiation.cc:407] Connection header received
0926 03:10:59.872929 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:59.872936 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:59.873017 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:59.873081 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:10:59.873912 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.875059 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.877528 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.878849 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.878998 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:59.879106 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:59.879490 (+ 384us) server_negotiation.cc:299] Negotiation successful
0926 03:10:59.879603 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 03:10:59.879956 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:59.871718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44263 (local address 127.0.97.2:48231)
0926 03:10:59.872565 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:59.872587 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:59.872750 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:59.873286 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:59.873291 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:59.873319 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:10:59.873743 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:59.873755 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.875222 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.875227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:59.877324 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:59.877346 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.879112 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.879118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:59.879290 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:59.879328 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 03:10:59.879444 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 03:10:59.879591 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":500,"threads_started":1}
I20260926 03:10:59.881304 19154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:59.871299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56773 (local address 127.0.97.3:39123)
0926 03:10:59.871486 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:59.871493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:59.874536 (+ 3043us) server_negotiation.cc:407] Connection header received
0926 03:10:59.874894 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:59.874900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:59.875009 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:59.875166 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:10:59.875819 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.876917 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.879094 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.880395 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.880534 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:59.880648 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:59.880994 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 03:10:59.881103 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 03:10:59.881175 18828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3578d5d53647678b20435a1e4c5516" candidate_uuid: "a1adcafff96246f8af5aa7f27ef2fd9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf24ac44026946698ac2f93cb5f895d8" is_pre_election: true
I20260926 03:10:59.881536 19168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:59.872015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39123 (local address 127.0.97.2:56773)
0926 03:10:59.874351 (+ 2336us) negotiation.cc:107] Waiting for socket to connect
0926 03:10:59.874371 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:10:59.874688 (+ 317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:10:59.875192 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:10:59.875197 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:10:59.875223 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:10:59.875664 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:59.875674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.877059 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.877063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:59.878933 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:10:59.878949 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.880610 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.880617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:10:59.880795 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:59.880828 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:10:59.880926 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 03:10:59.881056 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2165,"mutex_wait_us":20,"thread_start_us":1985,"threads_started":1}
I20260926 03:10:59.881800 18828 raft_consensus.cc:2454] T dd3578d5d53647678b20435a1e4c5516 P cf24ac44026946698ac2f93cb5f895d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1adcafff96246f8af5aa7f27ef2fd9a in term 0.
I20260926 03:10:59.882485 18898 leader_election.cc:302] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1adcafff96246f8af5aa7f27ef2fd9a, cf24ac44026946698ac2f93cb5f895d8; no voters:
I20260926 03:10:59.882587 19095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3578d5d53647678b20435a1e4c5516" candidate_uuid: "a1adcafff96246f8af5aa7f27ef2fd9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c36b7f5787e94a7494b5fabc9d73a233" is_pre_election: true
I20260926 03:10:59.882925 19163 raft_consensus.cc:2781] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:10:59.883073 19163 raft_consensus.cc:482] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:10:59.883177 19163 raft_consensus.cc:3037] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:10:59.883141 19095 raft_consensus.cc:2454] T dd3578d5d53647678b20435a1e4c5516 P c36b7f5787e94a7494b5fabc9d73a233 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1adcafff96246f8af5aa7f27ef2fd9a in term 0.
I20260926 03:10:59.885411 19163 raft_consensus.cc:504] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36b7f5787e94a7494b5fabc9d73a233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39123 } } peers { permanent_uuid: "a1adcafff96246f8af5aa7f27ef2fd9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33163 } } peers { permanent_uuid: "cf24ac44026946698ac2f93cb5f895d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44263 } }
I20260926 03:10:59.885988 19163 leader_election.cc:288] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a [CANDIDATE]: Term 1 election: Requested vote from peers c36b7f5787e94a7494b5fabc9d73a233 (127.0.97.3:39123), cf24ac44026946698ac2f93cb5f895d8 (127.0.97.1:44263)
I20260926 03:10:59.886538 18828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3578d5d53647678b20435a1e4c5516" candidate_uuid: "a1adcafff96246f8af5aa7f27ef2fd9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf24ac44026946698ac2f93cb5f895d8"
I20260926 03:10:59.886655 19095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd3578d5d53647678b20435a1e4c5516" candidate_uuid: "a1adcafff96246f8af5aa7f27ef2fd9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c36b7f5787e94a7494b5fabc9d73a233"
I20260926 03:10:59.886765 18828 raft_consensus.cc:3037] T dd3578d5d53647678b20435a1e4c5516 P cf24ac44026946698ac2f93cb5f895d8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:10:59.886893 19095 raft_consensus.cc:3037] T dd3578d5d53647678b20435a1e4c5516 P c36b7f5787e94a7494b5fabc9d73a233 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:10:59.889102 18828 raft_consensus.cc:2454] T dd3578d5d53647678b20435a1e4c5516 P cf24ac44026946698ac2f93cb5f895d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1adcafff96246f8af5aa7f27ef2fd9a in term 1.
I20260926 03:10:59.889101 19095 raft_consensus.cc:2454] T dd3578d5d53647678b20435a1e4c5516 P c36b7f5787e94a7494b5fabc9d73a233 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1adcafff96246f8af5aa7f27ef2fd9a in term 1.
I20260926 03:10:59.889604 18897 leader_election.cc:302] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1adcafff96246f8af5aa7f27ef2fd9a, c36b7f5787e94a7494b5fabc9d73a233; no voters:
I20260926 03:10:59.889968 19163 raft_consensus.cc:2781] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:10:59.890481 19163 raft_consensus.cc:686] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a [term 1 LEADER]: Becoming Leader. State: Replica: a1adcafff96246f8af5aa7f27ef2fd9a, State: Running, Role: LEADER
I20260926 03:10:59.891080 19163 consensus_queue.cc:237] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36b7f5787e94a7494b5fabc9d73a233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39123 } } peers { permanent_uuid: "a1adcafff96246f8af5aa7f27ef2fd9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33163 } } peers { permanent_uuid: "cf24ac44026946698ac2f93cb5f895d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44263 } }
I20260926 03:10:59.895565 18683 catalog_manager.cc:5495] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a reported cstate change: term changed from 0 to 1, leader changed from <none> to a1adcafff96246f8af5aa7f27ef2fd9a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "a1adcafff96246f8af5aa7f27ef2fd9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c36b7f5787e94a7494b5fabc9d73a233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1adcafff96246f8af5aa7f27ef2fd9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf24ac44026946698ac2f93cb5f895d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44263 } health_report { overall_health: UNKNOWN } } }
I20260926 03:10:59.922967 19155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:59.917256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44222 (local address 127.0.97.2:33163)
0926 03:10:59.917431 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:59.917436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:59.917544 (+ 108us) server_negotiation.cc:407] Connection header received
0926 03:10:59.917713 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:59.917717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:59.917798 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:59.917946 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:10:59.918357 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.919173 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.920656 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:10:59.921086 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:10:59.921190 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:10:59.921796 (+ 606us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:10:59.922288 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:10:59.922489 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 03:10:59.922707 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 03:10:59.922751 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 03:10:59.931521 18828 raft_consensus.cc:1261] T dd3578d5d53647678b20435a1e4c5516 P cf24ac44026946698ac2f93cb5f895d8 [term 1 FOLLOWER]: Refusing update from remote peer a1adcafff96246f8af5aa7f27ef2fd9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:10:59.931574 19095 raft_consensus.cc:1261] T dd3578d5d53647678b20435a1e4c5516 P c36b7f5787e94a7494b5fabc9d73a233 [term 1 FOLLOWER]: Refusing update from remote peer a1adcafff96246f8af5aa7f27ef2fd9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:10:59.932556 19163 consensus_queue.cc:1035] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf24ac44026946698ac2f93cb5f895d8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:10:59.932843 19169 consensus_queue.cc:1035] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c36b7f5787e94a7494b5fabc9d73a233" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:10:59.957767 18683 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:10:59.958083 18683 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:10:59.961948 18683 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 720ba0498b594cd980c029997a474ea0: Sending DeleteTablet for 3 replicas of tablet dd3578d5d53647678b20435a1e4c5516
I20260926 03:10:59.963306 18942 tablet_service.cc:1514] Processing DeleteTablet for tablet dd3578d5d53647678b20435a1e4c5516 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:10:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44210
I20260926 03:10:59.963301 19075 tablet_service.cc:1514] Processing DeleteTablet for tablet dd3578d5d53647678b20435a1e4c5516 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:10:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44668
I20260926 03:10:59.963414 18808 tablet_service.cc:1514] Processing DeleteTablet for tablet dd3578d5d53647678b20435a1e4c5516 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:10:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45426
I20260926 03:10:59.969251 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 18745
I20260926 03:10:59.998797 19181 tablet_replica.cc:330] T dd3578d5d53647678b20435a1e4c5516 P c36b7f5787e94a7494b5fabc9d73a233: stopping tablet replica
I20260926 03:11:00.005131 19181 raft_consensus.cc:2229] T dd3578d5d53647678b20435a1e4c5516 P c36b7f5787e94a7494b5fabc9d73a233 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:11:00.011230 19181 raft_consensus.cc:2258] T dd3578d5d53647678b20435a1e4c5516 P c36b7f5787e94a7494b5fabc9d73a233 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:11:00.012095 19182 tablet_replica.cc:330] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a: stopping tablet replica
I20260926 03:11:00.012872 19182 raft_consensus.cc:2229] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:11:00.013953 19182 raft_consensus.cc:2258] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:11:00.017920 19182 ts_tablet_manager.cc:1905] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:11:00.019806 19183 tablet_replica.cc:330] T dd3578d5d53647678b20435a1e4c5516 P cf24ac44026946698ac2f93cb5f895d8: stopping tablet replica
I20260926 03:11:00.020709 19183 raft_consensus.cc:2229] T dd3578d5d53647678b20435a1e4c5516 P cf24ac44026946698ac2f93cb5f895d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:11:00.021708 19181 ts_tablet_manager.cc:1905] T dd3578d5d53647678b20435a1e4c5516 P c36b7f5787e94a7494b5fabc9d73a233: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:11:00.022073 19183 raft_consensus.cc:2258] T dd3578d5d53647678b20435a1e4c5516 P cf24ac44026946698ac2f93cb5f895d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:11:00.024590 19183 ts_tablet_manager.cc:1905] T dd3578d5d53647678b20435a1e4c5516 P cf24ac44026946698ac2f93cb5f895d8: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 03:11:00 dist-test-slave-hmbw krb5kdc[18639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:11:00.030232 19181 ts_tablet_manager.cc:1918] T dd3578d5d53647678b20435a1e4c5516 P c36b7f5787e94a7494b5fabc9d73a233: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:11:00.030668 19181 log.cc:1196] T dd3578d5d53647678b20435a1e4c5516 P c36b7f5787e94a7494b5fabc9d73a233: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-2/wal/wals/dd3578d5d53647678b20435a1e4c5516
I20260926 03:11:00.031307 19181 ts_tablet_manager.cc:1939] T dd3578d5d53647678b20435a1e4c5516 P c36b7f5787e94a7494b5fabc9d73a233: Deleting consensus metadata
I20260926 03:11:00.034031 18670 catalog_manager.cc:4841] TS c36b7f5787e94a7494b5fabc9d73a233 (127.0.97.3:39123): tablet dd3578d5d53647678b20435a1e4c5516 (table test-table [id=f3047413908f442a90059b6419d46856]) successfully deleted
I20260926 03:11:00.036187 19183 ts_tablet_manager.cc:1918] T dd3578d5d53647678b20435a1e4c5516 P cf24ac44026946698ac2f93cb5f895d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:11:00.036545 19183 log.cc:1196] T dd3578d5d53647678b20435a1e4c5516 P cf24ac44026946698ac2f93cb5f895d8: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-0/wal/wals/dd3578d5d53647678b20435a1e4c5516
I20260926 03:11:00.037259 19182 ts_tablet_manager.cc:1918] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:11:00.037220 19183 ts_tablet_manager.cc:1939] T dd3578d5d53647678b20435a1e4c5516 P cf24ac44026946698ac2f93cb5f895d8: Deleting consensus metadata
I20260926 03:11:00.037501 19182 log.cc:1196] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391950058021-388-0/minicluster-data/ts-1/wal/wals/dd3578d5d53647678b20435a1e4c5516
I20260926 03:11:00.038154 19182 ts_tablet_manager.cc:1939] T dd3578d5d53647678b20435a1e4c5516 P a1adcafff96246f8af5aa7f27ef2fd9a: Deleting consensus metadata
I20260926 03:11:00.040132 18671 catalog_manager.cc:4841] TS cf24ac44026946698ac2f93cb5f895d8 (127.0.97.1:44263): tablet dd3578d5d53647678b20435a1e4c5516 (table test-table [id=f3047413908f442a90059b6419d46856]) successfully deleted
I20260926 03:11:00.040438 18671 catalog_manager.cc:4841] TS a1adcafff96246f8af5aa7f27ef2fd9a (127.0.97.2:33163): tablet dd3578d5d53647678b20435a1e4c5516 (table test-table [id=f3047413908f442a90059b6419d46856]) successfully deleted
I20260926 03:11:00.041307 19157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:10:59.970762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45436 (local address 127.0.97.1:44263)
0926 03:10:59.970958 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 03:10:59.970965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:10:59.995327 (+ 24362us) server_negotiation.cc:407] Connection header received
0926 03:10:59.995906 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:10:59.995914 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:10:59.996032 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:10:59.996117 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:00.005871 (+ 9754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:00.007725 (+ 1854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:00.019189 (+ 11464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:00.020272 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:00.020499 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:00.030895 (+ 10396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:00.030927 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:00.030963 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:00.031054 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:00.036094 (+ 5040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:00.037251 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:00.037389 (+ 138us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:00.037394 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:00.037820 (+ 426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:00.038848 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:00.038856 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:00.038860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:00.039454 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:00.039785 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:00.040624 (+ 839us) server_negotiation.cc:299] Negotiation successful
0926 03:11:00.040896 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:11:00.042099 18868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:11:00.133172 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 18745
I20260926 03:11:00.148789 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 18879
Sep 26 03:11:00 dist-test-slave-hmbw krb5kdc[18639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:11:00.167495 19155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:00.149584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44234 (local address 127.0.97.2:33163)
0926 03:11:00.149786 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:00.149792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:00.149819 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:11:00.149899 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:00.149905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:00.149985 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:00.150100 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:00.151501 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:00.152738 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:00.154266 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:00.155068 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:00.155208 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:00.160749 (+ 5541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:00.160774 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:00.160795 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:00.160846 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:00.164146 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:00.165132 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:00.165143 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:00.165146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:00.165436 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:00.165860 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:00.165863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:00.165865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:00.166231 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:00.166690 (+ 459us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:00.166913 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 03:11:00.167091 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:11:00.168347 19002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:11:00.282246 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 18879
I20260926 03:11:00.298408 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 19012
Sep 26 03:11:00 dist-test-slave-hmbw krb5kdc[18639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:11:00.315375 19154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:00.299206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44682 (local address 127.0.97.3:39123)
0926 03:11:00.299487 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:00.299492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:00.299530 (+ 38us) server_negotiation.cc:407] Connection header received
0926 03:11:00.299587 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:00.299590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:00.299667 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:00.299742 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:00.301130 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:00.302008 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:00.303482 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:00.303914 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:00.304018 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:00.308824 (+ 4806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:00.308844 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:00.308860 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:00.308912 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:00.311850 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:00.312822 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:00.312830 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:00.312833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:00.313015 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:00.313469 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:00.313474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:00.313477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:00.313891 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:00.314125 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:00.314873 (+ 748us) server_negotiation.cc:299] Negotiation successful
0926 03:11:00.315114 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 03:11:00.316059 19135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:11:00.406761 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 19012
I20260926 03:11:00.420827 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 18652
I20260926 03:11:00.421916 18714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:11:00.533658 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 18652
2026-09-26T03:11:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4218 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T03:11:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:11:00Z Disabled control of system clock
I20260926 03:11:00.617686 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39675
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:11:00.948123 19192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:00.948433 19192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:00.948537 19192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:00.955904 19192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:11:00.956025 19192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:00.956064 19192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:00.956144 19192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:11:00.956205 19192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:11:00.965786 19192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39675
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:00.967113 19192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:00.968907 19192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:00.981999 19197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:00.982016 19200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:00.982038 19198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:00.982801 19192 server_base.cc:1034] running on GCE node
I20260926 03:11:00.983644 19192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:00.985181 19192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:00.986423 19192 hybrid_clock.cc:648] HybridClock initialized: now 1790392260986356 us; error 58 us; skew 500 ppm
I20260926 03:11:00.986811 19192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:11:00.989220 19192 webserver.cc:460] Webserver started at http://127.0.97.62:44639/ using document root <none> and password file <none>
I20260926 03:11:00.989920 19192 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:00.990022 19192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:00.990387 19192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:00.993515 19192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "52c7c02fea9348febeb74fa7bb6ee2d4"
format_stamp: "Formatted at 2026-09-26 03:11:00 on dist-test-slave-hmbw"
server_key: "c8c54943f19bf9e78bad9b38bcfccf9f"
server_key_iv: "69fb570c1531930d4f297a7f5416c8d6"
server_key_version: "encryptionkey@0"
I20260926 03:11:00.994418 19192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "52c7c02fea9348febeb74fa7bb6ee2d4"
format_stamp: "Formatted at 2026-09-26 03:11:00 on dist-test-slave-hmbw"
server_key: "c8c54943f19bf9e78bad9b38bcfccf9f"
server_key_iv: "69fb570c1531930d4f297a7f5416c8d6"
server_key_version: "encryptionkey@0"
I20260926 03:11:01.000135 19192 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 03:11:01.004094 19206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:01.005960 19192 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 03:11:01.006191 19192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "52c7c02fea9348febeb74fa7bb6ee2d4"
format_stamp: "Formatted at 2026-09-26 03:11:00 on dist-test-slave-hmbw"
server_key: "c8c54943f19bf9e78bad9b38bcfccf9f"
server_key_iv: "69fb570c1531930d4f297a7f5416c8d6"
server_key_version: "encryptionkey@0"
I20260926 03:11:01.006431 19192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:01.029151 19192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:01.030015 19192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:01.030274 19192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:01.049312 19192 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39675
I20260926 03:11:01.049418 19257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39675 every 8 connection(s)
I20260926 03:11:01.050845 19192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:11:01.051769 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 19192
I20260926 03:11:01.051957 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:11:01.052444 388 external_mini_cluster.cc:1428] Setting key e2ef6369dbb1d3cda187b11296d6e5b5
I20260926 03:11:01.055909 19258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:01.069339 19261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:01.054344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54558 (local address 127.0.97.62:39675)
0926 03:11:01.055711 (+ 1367us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:01.055728 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:01.055781 (+ 53us) server_negotiation.cc:407] Connection header received
0926 03:11:01.058130 (+ 2349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:01.058160 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:01.058827 (+ 667us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:01.059414 (+ 587us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:01.060020 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:01.062089 (+ 2069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:01.063653 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:01.064721 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:01.064912 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:01.066281 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:01.066292 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:01.066318 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:01.067119 (+ 801us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:01.067229 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:01.067257 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:01.067397 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:01.067643 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 03:11:01.068032 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836,"thread_start_us":631,"threads_started":1}
I20260926 03:11:01.070235 19258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52c7c02fea9348febeb74fa7bb6ee2d4: Bootstrap starting.
I20260926 03:11:01.073335 19258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52c7c02fea9348febeb74fa7bb6ee2d4: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:01.074365 19258 log.cc:824] T 00000000000000000000000000000000 P 52c7c02fea9348febeb74fa7bb6ee2d4: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:01.077250 19258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52c7c02fea9348febeb74fa7bb6ee2d4: No bootstrap required, opened a new log
I20260926 03:11:01.085001 19258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52c7c02fea9348febeb74fa7bb6ee2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c7c02fea9348febeb74fa7bb6ee2d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39675 } }
I20260926 03:11:01.085400 19258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52c7c02fea9348febeb74fa7bb6ee2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:01.085506 19258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52c7c02fea9348febeb74fa7bb6ee2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c7c02fea9348febeb74fa7bb6ee2d4, State: Initialized, Role: FOLLOWER
I20260926 03:11:01.086092 19258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52c7c02fea9348febeb74fa7bb6ee2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c7c02fea9348febeb74fa7bb6ee2d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39675 } }
I20260926 03:11:01.086303 19258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52c7c02fea9348febeb74fa7bb6ee2d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:11:01.086436 19258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52c7c02fea9348febeb74fa7bb6ee2d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:11:01.086671 19258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52c7c02fea9348febeb74fa7bb6ee2d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:01.089075 19258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52c7c02fea9348febeb74fa7bb6ee2d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c7c02fea9348febeb74fa7bb6ee2d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39675 } }
I20260926 03:11:01.089632 19258 leader_election.cc:302] T 00000000000000000000000000000000 P 52c7c02fea9348febeb74fa7bb6ee2d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52c7c02fea9348febeb74fa7bb6ee2d4; no voters:
I20260926 03:11:01.090219 19258 leader_election.cc:288] T 00000000000000000000000000000000 P 52c7c02fea9348febeb74fa7bb6ee2d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:11:01.090371 19263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52c7c02fea9348febeb74fa7bb6ee2d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:11:01.091089 19263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52c7c02fea9348febeb74fa7bb6ee2d4 [term 1 LEADER]: Becoming Leader. State: Replica: 52c7c02fea9348febeb74fa7bb6ee2d4, State: Running, Role: LEADER
I20260926 03:11:01.091593 19263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52c7c02fea9348febeb74fa7bb6ee2d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c7c02fea9348febeb74fa7bb6ee2d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39675 } }
I20260926 03:11:01.092303 19258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52c7c02fea9348febeb74fa7bb6ee2d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:11:01.094110 19264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52c7c02fea9348febeb74fa7bb6ee2d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52c7c02fea9348febeb74fa7bb6ee2d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c7c02fea9348febeb74fa7bb6ee2d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39675 } } }
I20260926 03:11:01.094475 19264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52c7c02fea9348febeb74fa7bb6ee2d4 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:01.095319 19268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:11:01.095182 19265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52c7c02fea9348febeb74fa7bb6ee2d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52c7c02fea9348febeb74fa7bb6ee2d4. Latest consensus state: current_term: 1 leader_uuid: "52c7c02fea9348febeb74fa7bb6ee2d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52c7c02fea9348febeb74fa7bb6ee2d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39675 } } }
I20260926 03:11:01.095487 19265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52c7c02fea9348febeb74fa7bb6ee2d4 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:01.101197 19268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:11:01.111045 19268 catalog_manager.cc:1347] Generated new cluster ID: d950ffcff2194026bdf7540cdfa54e87
I20260926 03:11:01.111179 19268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:11:01.129383 19268 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:11:01.130759 19268 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:11:01.146761 19268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52c7c02fea9348febeb74fa7bb6ee2d4: Generated new TSK 0
I20260926 03:11:01.147837 19268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:11:01.167210 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39675
--builtin_ntp_servers=127.0.97.20:38999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:11:01.514396 19282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:01.514750 19282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:01.514883 19282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:01.522730 19282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:01.522854 19282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:01.522991 19282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:11:01.532923 19282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:01.534420 19282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:01.536309 19282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:01.549394 19287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:01.549446 19290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:01.549467 19288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:01.550271 19282 server_base.cc:1034] running on GCE node
I20260926 03:11:01.551026 19282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:01.552014 19282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:01.553246 19282 hybrid_clock.cc:648] HybridClock initialized: now 1790392261553190 us; error 47 us; skew 500 ppm
I20260926 03:11:01.553633 19282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:11:01.555982 19282 webserver.cc:460] Webserver started at http://127.0.97.1:42217/ using document root <none> and password file <none>
I20260926 03:11:01.556681 19282 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:01.556782 19282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:01.557087 19282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:01.560072 19282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-0/data/instance:
uuid: "6eac465b436f4604b2adb68ca932ecfb"
format_stamp: "Formatted at 2026-09-26 03:11:01 on dist-test-slave-hmbw"
server_key: "520c78ac5f7dab6646d2f995faa65780"
server_key_iv: "55ecd7474d3be4ba598fa6be78d33266"
server_key_version: "encryptionkey@0"
I20260926 03:11:01.560894 19282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6eac465b436f4604b2adb68ca932ecfb"
format_stamp: "Formatted at 2026-09-26 03:11:01 on dist-test-slave-hmbw"
server_key: "520c78ac5f7dab6646d2f995faa65780"
server_key_iv: "55ecd7474d3be4ba598fa6be78d33266"
server_key_version: "encryptionkey@0"
I20260926 03:11:01.566156 19282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:11:01.569737 19296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:01.571170 19282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 03:11:01.571339 19282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "6eac465b436f4604b2adb68ca932ecfb"
format_stamp: "Formatted at 2026-09-26 03:11:01 on dist-test-slave-hmbw"
server_key: "520c78ac5f7dab6646d2f995faa65780"
server_key_iv: "55ecd7474d3be4ba598fa6be78d33266"
server_key_version: "encryptionkey@0"
I20260926 03:11:01.571549 19282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:01.583596 19282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:01.584427 19282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:01.584681 19282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:01.585669 19282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:11:01.587280 19282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:11:01.587378 19282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:01.587483 19282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:11:01.587549 19282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:01.618242 19282 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40365
I20260926 03:11:01.618294 19408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40365 every 8 connection(s)
I20260926 03:11:01.619570 19282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:11:01.625216 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 19282
I20260926 03:11:01.625465 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:11:01.625937 388 external_mini_cluster.cc:1428] Setting key 782652867557814c6cf8d3bfd08c7daa
I20260926 03:11:01.629042 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39675
--builtin_ntp_servers=127.0.97.20:38999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:11:01.633419 19411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:01.622088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52501 (local address 127.0.97.62:39675)
0926 03:11:01.622628 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:01.622635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:01.623828 (+ 1193us) server_negotiation.cc:407] Connection header received
0926 03:11:01.625770 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:01.625778 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:01.625872 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:01.626004 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:01.627402 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:01.628408 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:01.630248 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:01.630818 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:01.630946 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:01.632539 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:01.632546 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:01.632551 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:01.632576 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:01.632623 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:01.632638 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:01.632724 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:01.632962 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 03:11:01.633065 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":261,"threads_started":1}
I20260926 03:11:01.634455 19412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:01.622692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39675 (local address 127.0.97.1:52501)
0926 03:11:01.623663 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:01.623706 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:01.625429 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:01.626229 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:01.626239 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:01.626351 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:11:01.627137 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:01.627164 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:01.628598 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:01.628603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:01.630089 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:01.630104 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:01.631062 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:01.631067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:01.631367 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:01.632098 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:11:01.632101 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:11:01.632153 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:11:01.632246 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:11:01.632751 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:11:01.632761 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:11:01.632768 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 03:11:01.632950 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 03:11:01.633290 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":298,"threads_started":1}
I20260926 03:11:01.636269 19409 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39675
I20260926 03:11:01.636592 19409 heartbeater.cc:464] Registering TS with master...
I20260926 03:11:01.637348 19409 heartbeater.cc:511] Master 127.0.97.62:39675 requested a full tablet report, sending...
I20260926 03:11:01.639598 19223 ts_manager.cc:194] Registered new tserver with Master: 6eac465b436f4604b2adb68ca932ecfb (127.0.97.1:40365)
I20260926 03:11:01.641132 19223 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:52501
W20260926 03:11:01.964052 19413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:01.964356 19413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:01.964468 19413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:01.972033 19413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:01.972151 19413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:01.972286 19413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:11:01.982296 19413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:01.983707 19413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:01.985778 19413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:02.001186 19418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:02.001165 19419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:02.001124 19421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:02.002250 19413 server_base.cc:1034] running on GCE node
I20260926 03:11:02.002998 19413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:02.003983 19413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:02.005236 19413 hybrid_clock.cc:648] HybridClock initialized: now 1790392262005174 us; error 52 us; skew 500 ppm
I20260926 03:11:02.005659 19413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:11:02.008158 19413 webserver.cc:460] Webserver started at http://127.0.97.2:36157/ using document root <none> and password file <none>
I20260926 03:11:02.009068 19413 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:02.009197 19413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:02.009568 19413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:02.013569 19413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-1/data/instance:
uuid: "c3ddbe3956a148f597f88da0aef3de4e"
format_stamp: "Formatted at 2026-09-26 03:11:02 on dist-test-slave-hmbw"
server_key: "ffc04e8e5f16a0fdcc617e2ea2d749cf"
server_key_iv: "8ca9d6b6c66526461b6f152666ef21d4"
server_key_version: "encryptionkey@0"
I20260926 03:11:02.014489 19413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c3ddbe3956a148f597f88da0aef3de4e"
format_stamp: "Formatted at 2026-09-26 03:11:02 on dist-test-slave-hmbw"
server_key: "ffc04e8e5f16a0fdcc617e2ea2d749cf"
server_key_iv: "8ca9d6b6c66526461b6f152666ef21d4"
server_key_version: "encryptionkey@0"
I20260926 03:11:02.020241 19413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:11:02.024295 19427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:02.026156 19413 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260926 03:11:02.026364 19413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "c3ddbe3956a148f597f88da0aef3de4e"
format_stamp: "Formatted at 2026-09-26 03:11:02 on dist-test-slave-hmbw"
server_key: "ffc04e8e5f16a0fdcc617e2ea2d749cf"
server_key_iv: "8ca9d6b6c66526461b6f152666ef21d4"
server_key_version: "encryptionkey@0"
I20260926 03:11:02.026744 19413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:02.043098 19413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:02.043977 19413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:02.044243 19413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:02.045317 19413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:11:02.047037 19413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:11:02.047139 19413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:02.047254 19413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:11:02.047354 19413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:02.082257 19413 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42011
I20260926 03:11:02.082322 19539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42011 every 8 connection(s)
I20260926 03:11:02.083673 19413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:11:02.085135 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 19413
I20260926 03:11:02.085358 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:11:02.085929 388 external_mini_cluster.cc:1428] Setting key d5ea64a4753c8ad7e64b540488fd63e5
I20260926 03:11:02.088766 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39675
--builtin_ntp_servers=127.0.97.20:38999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:11:02.098572 19411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:02.086566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35801 (local address 127.0.97.62:39675)
0926 03:11:02.086790 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:02.086796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:02.088411 (+ 1615us) server_negotiation.cc:407] Connection header received
0926 03:11:02.090397 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:02.090403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:02.090532 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:02.090632 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:02.091922 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.092943 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.095257 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.095696 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.095871 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:02.097595 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:02.097603 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:02.097608 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:02.097634 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:02.097677 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:02.097693 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:02.097798 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:02.098061 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 03:11:02.098193 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 03:11:02.099776 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:02.087170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39675 (local address 127.0.97.2:35801)
0926 03:11:02.088255 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:02.088299 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:02.090075 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:02.090968 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:02.090981 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:02.091094 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:11:02.091712 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:02.091732 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.093151 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.093156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:02.095023 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:02.095045 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.096010 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.096018 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:02.096261 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:02.097113 (+ 852us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:11:02.097116 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:11:02.097171 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:11:02.097296 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:11:02.097842 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:11:02.097852 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:11:02.097859 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 03:11:02.098044 (+ 185us) client_negotiation.cc:240] Negotiation successful
0926 03:11:02.098475 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":337,"threads_started":1}
I20260926 03:11:02.101369 19540 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39675
I20260926 03:11:02.101707 19540 heartbeater.cc:464] Registering TS with master...
I20260926 03:11:02.102811 19540 heartbeater.cc:511] Master 127.0.97.62:39675 requested a full tablet report, sending...
I20260926 03:11:02.104455 19223 ts_manager.cc:194] Registered new tserver with Master: c3ddbe3956a148f597f88da0aef3de4e (127.0.97.2:42011)
I20260926 03:11:02.105341 19223 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:35801
W20260926 03:11:02.433012 19543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:02.433288 19543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:02.433413 19543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:02.440690 19543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:02.440800 19543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:02.440944 19543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:11:02.450194 19543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:02.451558 19543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:02.453306 19543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:02.466536 19548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:02.466603 19549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:02.466567 19551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:02.467478 19543 server_base.cc:1034] running on GCE node
I20260926 03:11:02.468117 19543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:02.469152 19543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:02.470409 19543 hybrid_clock.cc:648] HybridClock initialized: now 1790392262470347 us; error 51 us; skew 500 ppm
I20260926 03:11:02.470862 19543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:11:02.473564 19543 webserver.cc:460] Webserver started at http://127.0.97.3:41691/ using document root <none> and password file <none>
I20260926 03:11:02.474370 19543 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:02.474485 19543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:02.474920 19543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:02.478559 19543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-2/data/instance:
uuid: "23f9933d61c6400080f7e947702154bb"
format_stamp: "Formatted at 2026-09-26 03:11:02 on dist-test-slave-hmbw"
server_key: "161ac7fef38888693cd8c381796cbc91"
server_key_iv: "e104c70b22bda6d7a80ff4f4acc1334d"
server_key_version: "encryptionkey@0"
I20260926 03:11:02.479501 19543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-2/wal/instance:
uuid: "23f9933d61c6400080f7e947702154bb"
format_stamp: "Formatted at 2026-09-26 03:11:02 on dist-test-slave-hmbw"
server_key: "161ac7fef38888693cd8c381796cbc91"
server_key_iv: "e104c70b22bda6d7a80ff4f4acc1334d"
server_key_version: "encryptionkey@0"
I20260926 03:11:02.485414 19543 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 03:11:02.489094 19557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:02.490813 19543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 03:11:02.490985 19543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "23f9933d61c6400080f7e947702154bb"
format_stamp: "Formatted at 2026-09-26 03:11:02 on dist-test-slave-hmbw"
server_key: "161ac7fef38888693cd8c381796cbc91"
server_key_iv: "e104c70b22bda6d7a80ff4f4acc1334d"
server_key_version: "encryptionkey@0"
I20260926 03:11:02.491221 19543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:02.511525 19543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:02.512393 19543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:02.512674 19543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:02.513800 19543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:11:02.515538 19543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:11:02.515636 19543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:02.515745 19543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:11:02.515821 19543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:02.546662 19543 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44105
I20260926 03:11:02.546670 19669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44105 every 8 connection(s)
I20260926 03:11:02.548056 19543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:11:02.555678 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 19543
I20260926 03:11:02.555855 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:11:02.556272 388 external_mini_cluster.cc:1428] Setting key 3c30edd4d9a2a24316f2e9ab534696bb
I20260926 03:11:02.561090 19411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:02.550651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55749 (local address 127.0.97.62:39675)
0926 03:11:02.550877 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:02.550883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:02.552379 (+ 1496us) server_negotiation.cc:407] Connection header received
0926 03:11:02.554229 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:02.554234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:02.554328 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:02.554424 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:02.555676 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.556514 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.558343 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.558875 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.559019 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:02.560369 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:02.560430 (+ 61us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:02.560434 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:02.560453 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:02.560486 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:02.560499 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:02.560570 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:02.560744 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 03:11:02.560826 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:11:02.562143 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:02.551253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39675 (local address 127.0.97.3:55749)
0926 03:11:02.552215 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:02.552254 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:02.553925 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:02.554700 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:02.554709 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:02.554805 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:11:02.555477 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:02.555503 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.556668 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.556673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:02.558203 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:02.558219 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.559119 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.559127 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:02.559399 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:02.560046 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:11:02.560048 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:11:02.560090 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:11:02.560187 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:11:02.560578 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:11:02.560584 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:11:02.560590 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 03:11:02.560731 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 03:11:02.561085 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":294,"threads_started":1}
I20260926 03:11:02.563609 19670 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39675
I20260926 03:11:02.563894 19670 heartbeater.cc:464] Registering TS with master...
I20260926 03:11:02.564626 19670 heartbeater.cc:511] Master 127.0.97.62:39675 requested a full tablet report, sending...
I20260926 03:11:02.566118 19223 ts_manager.cc:194] Registered new tserver with Master: 23f9933d61c6400080f7e947702154bb (127.0.97.3:44105)
I20260926 03:11:02.567260 19223 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:55749
I20260926 03:11:02.571766 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:11:02.582901 19411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:02.576349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54572 (local address 127.0.97.62:39675)
0926 03:11:02.576556 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:02.576563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:02.577031 (+ 468us) server_negotiation.cc:407] Connection header received
0926 03:11:02.577198 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:02.577204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:02.577275 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:02.577360 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:02.577827 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.578670 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.580356 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.580903 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.581016 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:02.582150 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:02.582155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:02.582158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:02.582177 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:02.582208 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:02.582219 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:02.582330 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:02.582613 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 03:11:02.582665 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:11:02.587100 19223 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:11:02.594442 19223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:11:02.627614 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:02.607297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54258 (local address 127.0.97.2:42011)
0926 03:11:02.609005 (+ 1708us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:02.609013 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:02.611675 (+ 2662us) server_negotiation.cc:407] Connection header received
0926 03:11:02.612105 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:02.612117 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:02.612534 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:02.612714 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:11:02.613791 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.616992 (+ 3201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.622418 (+ 5426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.625723 (+ 3305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.625846 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:02.626105 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:02.626991 (+ 886us) server_negotiation.cc:299] Negotiation successful
0926 03:11:02.627219 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1571,"thread_start_us":1470,"threads_started":1}
I20260926 03:11:02.628547 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:02.607265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42011 (local address 127.0.0.1:54258)
0926 03:11:02.610066 (+ 2801us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:02.611555 (+ 1489us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:02.611854 (+ 299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:02.613062 (+ 1208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:02.613071 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:02.613106 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:11:02.613562 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:02.613576 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.617236 (+ 3660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.617243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:02.621399 (+ 4156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:02.621420 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.626314 (+ 4894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.626321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:02.626675 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:02.626746 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 03:11:02.627857 (+ 1111us) client_negotiation.cc:240] Negotiation successful
0926 03:11:02.628108 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2516,"mutex_wait_us":1,"thread_start_us":14454,"threads_started":1}
I20260926 03:11:02.632382 19684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:02.607332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40365 (local address 127.0.0.1:45618)
0926 03:11:02.622048 (+ 14716us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:02.622071 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:02.622222 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:02.623183 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:02.623188 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:02.623217 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:11:02.623619 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:02.623634 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.625623 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.625629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:02.628332 (+ 2703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:02.628352 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.631398 (+ 3046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.631405 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:02.631693 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:02.631728 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 03:11:02.631858 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 03:11:02.631993 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14561,"mutex_wait_us":42,"thread_start_us":2284,"threads_started":1}
I20260926 03:11:02.632453 19685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:02.607833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45618 (local address 127.0.97.1:40365)
0926 03:11:02.608578 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:02.608589 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:02.622193 (+ 13604us) server_negotiation.cc:407] Connection header received
0926 03:11:02.622363 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:02.622372 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:02.622701 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:02.622896 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:11:02.623816 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.625392 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.628597 (+ 3205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.630179 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.630292 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:02.630595 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:02.631950 (+ 1355us) server_negotiation.cc:299] Negotiation successful
0926 03:11:02.632124 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":349,"threads_started":1}
I20260926 03:11:02.633823 19475 tablet_service.cc:1467] Processing CreateTablet for tablet a35d94ca9bcc45cd92872dc8364f5520 (DEFAULT_TABLE table=test-table [id=92e316ad597a42a7bd22ff1755470a50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:11:02.635267 19475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a35d94ca9bcc45cd92872dc8364f5520. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:02.635658 19344 tablet_service.cc:1467] Processing CreateTablet for tablet a35d94ca9bcc45cd92872dc8364f5520 (DEFAULT_TABLE table=test-table [id=92e316ad597a42a7bd22ff1755470a50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:11:02.636951 19344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a35d94ca9bcc45cd92872dc8364f5520. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:02.639029 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:02.609848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44105 (local address 127.0.0.1:46072)
0926 03:11:02.628816 (+ 18968us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:02.628838 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:02.628951 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:02.632071 (+ 3120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:02.632076 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:02.632104 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:11:02.632280 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:02.632290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.634726 (+ 2436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.634731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:02.636486 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:02.636502 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.638229 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.638236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:02.638450 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:02.638488 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 03:11:02.638596 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 03:11:02.638731 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18856,"thread_start_us":20746,"threads_started":1}
I20260926 03:11:02.640136 19687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:02.610669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46072 (local address 127.0.97.3:44105)
0926 03:11:02.631224 (+ 20555us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:02.631233 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:02.631279 (+ 46us) server_negotiation.cc:407] Connection header received
0926 03:11:02.631386 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:02.631394 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:02.631643 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:02.632463 (+ 820us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:11:02.632903 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.634566 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.636670 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.638038 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.639286 (+ 1248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:02.639548 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:02.639619 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 03:11:02.639787 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20377,"thread_start_us":21134,"threads_started":1}
I20260926 03:11:02.644368 19409 heartbeater.cc:503] Master 127.0.97.62:39675 was elected leader, sending a full tablet report...
I20260926 03:11:02.644033 19605 tablet_service.cc:1467] Processing CreateTablet for tablet a35d94ca9bcc45cd92872dc8364f5520 (DEFAULT_TABLE table=test-table [id=92e316ad597a42a7bd22ff1755470a50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:11:02.645495 19605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a35d94ca9bcc45cd92872dc8364f5520. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:02.649040 19688 tablet_bootstrap.cc:492] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb: Bootstrap starting.
I20260926 03:11:02.650005 19689 tablet_bootstrap.cc:492] T a35d94ca9bcc45cd92872dc8364f5520 P c3ddbe3956a148f597f88da0aef3de4e: Bootstrap starting.
I20260926 03:11:02.653601 19689 tablet_bootstrap.cc:654] T a35d94ca9bcc45cd92872dc8364f5520 P c3ddbe3956a148f597f88da0aef3de4e: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:02.653601 19688 tablet_bootstrap.cc:654] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:02.654904 19689 log.cc:824] T a35d94ca9bcc45cd92872dc8364f5520 P c3ddbe3956a148f597f88da0aef3de4e: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:02.654912 19688 log.cc:824] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:02.657691 19688 tablet_bootstrap.cc:492] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb: No bootstrap required, opened a new log
I20260926 03:11:02.657980 19688 ts_tablet_manager.cc:1397] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260926 03:11:02.658201 19689 tablet_bootstrap.cc:492] T a35d94ca9bcc45cd92872dc8364f5520 P c3ddbe3956a148f597f88da0aef3de4e: No bootstrap required, opened a new log
I20260926 03:11:02.658581 19689 ts_tablet_manager.cc:1397] T a35d94ca9bcc45cd92872dc8364f5520 P c3ddbe3956a148f597f88da0aef3de4e: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260926 03:11:02.659885 19692 tablet_bootstrap.cc:492] T a35d94ca9bcc45cd92872dc8364f5520 P 23f9933d61c6400080f7e947702154bb: Bootstrap starting.
I20260926 03:11:02.664726 19692 tablet_bootstrap.cc:654] T a35d94ca9bcc45cd92872dc8364f5520 P 23f9933d61c6400080f7e947702154bb: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:02.666117 19692 log.cc:824] T a35d94ca9bcc45cd92872dc8364f5520 P 23f9933d61c6400080f7e947702154bb: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:02.666086 19689 raft_consensus.cc:348] T a35d94ca9bcc45cd92872dc8364f5520 P c3ddbe3956a148f597f88da0aef3de4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ddbe3956a148f597f88da0aef3de4e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } } peers { permanent_uuid: "6eac465b436f4604b2adb68ca932ecfb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40365 } } peers { permanent_uuid: "23f9933d61c6400080f7e947702154bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44105 } }
I20260926 03:11:02.666539 19689 raft_consensus.cc:374] T a35d94ca9bcc45cd92872dc8364f5520 P c3ddbe3956a148f597f88da0aef3de4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:02.666646 19689 raft_consensus.cc:729] T a35d94ca9bcc45cd92872dc8364f5520 P c3ddbe3956a148f597f88da0aef3de4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3ddbe3956a148f597f88da0aef3de4e, State: Initialized, Role: FOLLOWER
I20260926 03:11:02.667424 19689 consensus_queue.cc:260] T a35d94ca9bcc45cd92872dc8364f5520 P c3ddbe3956a148f597f88da0aef3de4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ddbe3956a148f597f88da0aef3de4e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } } peers { permanent_uuid: "6eac465b436f4604b2adb68ca932ecfb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40365 } } peers { permanent_uuid: "23f9933d61c6400080f7e947702154bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44105 } }
I20260926 03:11:02.667927 19688 raft_consensus.cc:348] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ddbe3956a148f597f88da0aef3de4e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } } peers { permanent_uuid: "6eac465b436f4604b2adb68ca932ecfb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40365 } } peers { permanent_uuid: "23f9933d61c6400080f7e947702154bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44105 } }
I20260926 03:11:02.668349 19688 raft_consensus.cc:374] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:02.668540 19688 raft_consensus.cc:729] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eac465b436f4604b2adb68ca932ecfb, State: Initialized, Role: FOLLOWER
I20260926 03:11:02.668579 19540 heartbeater.cc:503] Master 127.0.97.62:39675 was elected leader, sending a full tablet report...
I20260926 03:11:02.669142 19688 consensus_queue.cc:260] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ddbe3956a148f597f88da0aef3de4e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } } peers { permanent_uuid: "6eac465b436f4604b2adb68ca932ecfb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40365 } } peers { permanent_uuid: "23f9933d61c6400080f7e947702154bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44105 } }
I20260926 03:11:02.669596 19692 tablet_bootstrap.cc:492] T a35d94ca9bcc45cd92872dc8364f5520 P 23f9933d61c6400080f7e947702154bb: No bootstrap required, opened a new log
I20260926 03:11:02.670534 19689 ts_tablet_manager.cc:1428] T a35d94ca9bcc45cd92872dc8364f5520 P c3ddbe3956a148f597f88da0aef3de4e: Time spent starting tablet: real 0.012s user 0.009s sys 0.000s
I20260926 03:11:02.670871 19688 ts_tablet_manager.cc:1428] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260926 03:11:02.671500 19692 ts_tablet_manager.cc:1397] T a35d94ca9bcc45cd92872dc8364f5520 P 23f9933d61c6400080f7e947702154bb: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260926 03:11:02.676306 19695 raft_consensus.cc:482] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:11:02.676578 19695 raft_consensus.cc:504] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ddbe3956a148f597f88da0aef3de4e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } } peers { permanent_uuid: "6eac465b436f4604b2adb68ca932ecfb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40365 } } peers { permanent_uuid: "23f9933d61c6400080f7e947702154bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44105 } }
I20260926 03:11:02.678145 19695 leader_election.cc:288] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3ddbe3956a148f597f88da0aef3de4e (127.0.97.2:42011), 23f9933d61c6400080f7e947702154bb (127.0.97.3:44105)
I20260926 03:11:02.682484 19692 raft_consensus.cc:348] T a35d94ca9bcc45cd92872dc8364f5520 P 23f9933d61c6400080f7e947702154bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ddbe3956a148f597f88da0aef3de4e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } } peers { permanent_uuid: "6eac465b436f4604b2adb68ca932ecfb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40365 } } peers { permanent_uuid: "23f9933d61c6400080f7e947702154bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44105 } }
I20260926 03:11:02.683048 19692 raft_consensus.cc:374] T a35d94ca9bcc45cd92872dc8364f5520 P 23f9933d61c6400080f7e947702154bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:02.683151 19692 raft_consensus.cc:729] T a35d94ca9bcc45cd92872dc8364f5520 P 23f9933d61c6400080f7e947702154bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23f9933d61c6400080f7e947702154bb, State: Initialized, Role: FOLLOWER
I20260926 03:11:02.683756 19692 consensus_queue.cc:260] T a35d94ca9bcc45cd92872dc8364f5520 P 23f9933d61c6400080f7e947702154bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ddbe3956a148f597f88da0aef3de4e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } } peers { permanent_uuid: "6eac465b436f4604b2adb68ca932ecfb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40365 } } peers { permanent_uuid: "23f9933d61c6400080f7e947702154bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44105 } }
I20260926 03:11:02.690150 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:02.678774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41155 (local address 127.0.97.2:42011)
0926 03:11:02.678969 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:02.678976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:02.683269 (+ 4293us) server_negotiation.cc:407] Connection header received
0926 03:11:02.684149 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:02.684154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:02.684258 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:02.684380 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:11:02.685143 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.686020 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.688229 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.689164 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.689245 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:02.689338 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:02.689761 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 03:11:02.689901 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 03:11:02.690441 19696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:02.678351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42011 (local address 127.0.97.1:41155)
0926 03:11:02.683124 (+ 4773us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:02.683148 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:02.684017 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:02.684565 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:02.684571 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:02.684602 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:11:02.684978 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:02.684991 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.686157 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.686163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:02.687921 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:02.687939 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.689480 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.689484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:02.689606 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:02.689629 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 03:11:02.689708 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 03:11:02.689843 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4612,"mutex_wait_us":1,"thread_start_us":4381,"threads_started":1}
I20260926 03:11:02.690997 19670 heartbeater.cc:503] Master 127.0.97.62:39675 was elected leader, sending a full tablet report...
I20260926 03:11:02.691450 19495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35d94ca9bcc45cd92872dc8364f5520" candidate_uuid: "6eac465b436f4604b2adb68ca932ecfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3ddbe3956a148f597f88da0aef3de4e" is_pre_election: true
I20260926 03:11:02.692032 19495 raft_consensus.cc:2454] T a35d94ca9bcc45cd92872dc8364f5520 P c3ddbe3956a148f597f88da0aef3de4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6eac465b436f4604b2adb68ca932ecfb in term 0.
I20260926 03:11:02.692385 19692 ts_tablet_manager.cc:1428] T a35d94ca9bcc45cd92872dc8364f5520 P 23f9933d61c6400080f7e947702154bb: Time spent starting tablet: real 0.021s user 0.014s sys 0.005s
I20260926 03:11:02.692643 19687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:02.678642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42053 (local address 127.0.97.3:44105)
0926 03:11:02.678969 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:02.678975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:02.683809 (+ 4834us) server_negotiation.cc:407] Connection header received
0926 03:11:02.684006 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:02.684012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:02.684090 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:02.684166 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:11:02.684644 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.685571 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.688481 (+ 2910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.689852 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.689960 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:02.690088 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:02.692281 (+ 2193us) server_negotiation.cc:299] Negotiation successful
0926 03:11:02.692408 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260926 03:11:02.693030 19697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:02.678427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44105 (local address 127.0.97.1:42053)
0926 03:11:02.683705 (+ 5278us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:02.683721 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:02.683874 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:02.684178 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:02.684182 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:02.684205 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:11:02.684494 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:02.684504 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.685702 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.685708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:02.688205 (+ 2497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:02.688224 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.690389 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.690397 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:02.690606 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:02.690640 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 03:11:02.690738 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 03:11:02.690872 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5171,"mutex_wait_us":1,"thread_start_us":5004,"threads_started":1}
I20260926 03:11:02.692937 19300 leader_election.cc:302] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6eac465b436f4604b2adb68ca932ecfb, c3ddbe3956a148f597f88da0aef3de4e; no voters:
I20260926 03:11:02.693466 19695 raft_consensus.cc:2781] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:11:02.693627 19695 raft_consensus.cc:482] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:11:02.693734 19695 raft_consensus.cc:3037] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:02.694020 19625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35d94ca9bcc45cd92872dc8364f5520" candidate_uuid: "6eac465b436f4604b2adb68ca932ecfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23f9933d61c6400080f7e947702154bb" is_pre_election: true
I20260926 03:11:02.694701 19625 raft_consensus.cc:2454] T a35d94ca9bcc45cd92872dc8364f5520 P 23f9933d61c6400080f7e947702154bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6eac465b436f4604b2adb68ca932ecfb in term 0.
I20260926 03:11:02.697012 19695 raft_consensus.cc:504] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ddbe3956a148f597f88da0aef3de4e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } } peers { permanent_uuid: "6eac465b436f4604b2adb68ca932ecfb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40365 } } peers { permanent_uuid: "23f9933d61c6400080f7e947702154bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44105 } }
I20260926 03:11:02.697939 19695 leader_election.cc:288] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb [CANDIDATE]: Term 1 election: Requested vote from peers c3ddbe3956a148f597f88da0aef3de4e (127.0.97.2:42011), 23f9933d61c6400080f7e947702154bb (127.0.97.3:44105)
I20260926 03:11:02.698390 19625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35d94ca9bcc45cd92872dc8364f5520" candidate_uuid: "6eac465b436f4604b2adb68ca932ecfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23f9933d61c6400080f7e947702154bb"
I20260926 03:11:02.698381 19495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35d94ca9bcc45cd92872dc8364f5520" candidate_uuid: "6eac465b436f4604b2adb68ca932ecfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3ddbe3956a148f597f88da0aef3de4e"
I20260926 03:11:02.698618 19625 raft_consensus.cc:3037] T a35d94ca9bcc45cd92872dc8364f5520 P 23f9933d61c6400080f7e947702154bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:02.698664 19495 raft_consensus.cc:3037] T a35d94ca9bcc45cd92872dc8364f5520 P c3ddbe3956a148f597f88da0aef3de4e [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:02.700984 19625 raft_consensus.cc:2454] T a35d94ca9bcc45cd92872dc8364f5520 P 23f9933d61c6400080f7e947702154bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6eac465b436f4604b2adb68ca932ecfb in term 1.
I20260926 03:11:02.701074 19495 raft_consensus.cc:2454] T a35d94ca9bcc45cd92872dc8364f5520 P c3ddbe3956a148f597f88da0aef3de4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6eac465b436f4604b2adb68ca932ecfb in term 1.
I20260926 03:11:02.701486 19299 leader_election.cc:302] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23f9933d61c6400080f7e947702154bb, 6eac465b436f4604b2adb68ca932ecfb; no voters:
I20260926 03:11:02.701850 19695 raft_consensus.cc:2781] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:11:02.702422 19695 raft_consensus.cc:686] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb [term 1 LEADER]: Becoming Leader. State: Replica: 6eac465b436f4604b2adb68ca932ecfb, State: Running, Role: LEADER
I20260926 03:11:02.702924 19695 consensus_queue.cc:237] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ddbe3956a148f597f88da0aef3de4e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } } peers { permanent_uuid: "6eac465b436f4604b2adb68ca932ecfb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40365 } } peers { permanent_uuid: "23f9933d61c6400080f7e947702154bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44105 } }
I20260926 03:11:02.708022 19223 catalog_manager.cc:5495] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6eac465b436f4604b2adb68ca932ecfb (127.0.97.1). New cstate: current_term: 1 leader_uuid: "6eac465b436f4604b2adb68ca932ecfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ddbe3956a148f597f88da0aef3de4e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6eac465b436f4604b2adb68ca932ecfb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23f9933d61c6400080f7e947702154bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44105 } health_report { overall_health: UNKNOWN } } }
I20260926 03:11:02.745244 19685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:02.739970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45634 (local address 127.0.97.1:40365)
0926 03:11:02.740158 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:02.740164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:02.740183 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:11:02.740315 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:02.740319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:02.740414 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:02.740541 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:11:02.740866 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.741632 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.743051 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.743492 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.743592 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:02.744125 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:11:02.744626 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:11:02.744757 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:02.744929 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 03:11:02.744969 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 03:11:02.754236 19625 raft_consensus.cc:1261] T a35d94ca9bcc45cd92872dc8364f5520 P 23f9933d61c6400080f7e947702154bb [term 1 FOLLOWER]: Refusing update from remote peer 6eac465b436f4604b2adb68ca932ecfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:11:02.754237 19495 raft_consensus.cc:1261] T a35d94ca9bcc45cd92872dc8364f5520 P c3ddbe3956a148f597f88da0aef3de4e [term 1 FOLLOWER]: Refusing update from remote peer 6eac465b436f4604b2adb68ca932ecfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:11:02.755384 19695 consensus_queue.cc:1035] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "23f9933d61c6400080f7e947702154bb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:11:02.755734 19699 consensus_queue.cc:1035] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3ddbe3956a148f597f88da0aef3de4e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:11:02.769510 19704 mvcc.cc:205] Tried to move back new op lower bound from 7333446708230615040 to 7333446708035014656. Current Snapshot: MvccSnapshot[applied={T|T < 7333446708230615040}]
I20260926 03:11:02.781744 19221 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:11:02.782019 19221 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:11:02.786171 19221 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52c7c02fea9348febeb74fa7bb6ee2d4: Sending DeleteTablet for 3 replicas of tablet a35d94ca9bcc45cd92872dc8364f5520
I20260926 03:11:02.787472 19475 tablet_service.cc:1514] Processing DeleteTablet for tablet a35d94ca9bcc45cd92872dc8364f5520 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:11:02 UTC) from {username='slave'} at 127.0.0.1:54258
I20260926 03:11:02.787631 19605 tablet_service.cc:1514] Processing DeleteTablet for tablet a35d94ca9bcc45cd92872dc8364f5520 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:11:02 UTC) from {username='slave'} at 127.0.0.1:46072
I20260926 03:11:02.787864 19344 tablet_service.cc:1514] Processing DeleteTablet for tablet a35d94ca9bcc45cd92872dc8364f5520 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:11:02 UTC) from {username='slave'} at 127.0.0.1:45618
I20260926 03:11:02.789752 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 19282
I20260926 03:11:02.807031 19712 tablet_replica.cc:330] T a35d94ca9bcc45cd92872dc8364f5520 P c3ddbe3956a148f597f88da0aef3de4e: stopping tablet replica
I20260926 03:11:02.807654 19712 raft_consensus.cc:2229] T a35d94ca9bcc45cd92872dc8364f5520 P c3ddbe3956a148f597f88da0aef3de4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:11:02.808348 19712 raft_consensus.cc:2258] T a35d94ca9bcc45cd92872dc8364f5520 P c3ddbe3956a148f597f88da0aef3de4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:11:02.811610 19711 tablet_replica.cc:330] T a35d94ca9bcc45cd92872dc8364f5520 P 23f9933d61c6400080f7e947702154bb: stopping tablet replica
I20260926 03:11:02.810987 19712 ts_tablet_manager.cc:1905] T a35d94ca9bcc45cd92872dc8364f5520 P c3ddbe3956a148f597f88da0aef3de4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:11:02.812712 19713 tablet_replica.cc:330] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb: stopping tablet replica
I20260926 03:11:02.813122 19713 raft_consensus.cc:2229] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:11:02.814457 19685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:02.791060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45638 (local address 127.0.97.1:40365)
0926 03:11:02.791244 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:02.791250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:02.802041 (+ 10791us) server_negotiation.cc:407] Connection header received
0926 03:11:02.802271 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:02.802277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:02.802429 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:02.802595 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:02.806229 (+ 3634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.807798 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.810008 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.810840 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.810975 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:02.812550 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:02.812563 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:02.812582 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:02.813400 (+ 818us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:02.813472 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:02.813496 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:02.813605 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:02.814056 (+ 451us) server_negotiation.cc:299] Negotiation successful
0926 03:11:02.814164 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:11:02.812083 19711 raft_consensus.cc:2229] T a35d94ca9bcc45cd92872dc8364f5520 P 23f9933d61c6400080f7e947702154bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:11:02.814985 19713 raft_consensus.cc:2258] T a35d94ca9bcc45cd92872dc8364f5520 P 6eac465b436f4604b2adb68ca932ecfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:11:02.815284 19404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:11:02.815796 19711 raft_consensus.cc:2258] T a35d94ca9bcc45cd92872dc8364f5520 P 23f9933d61c6400080f7e947702154bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:11:02.818109 19711 ts_tablet_manager.cc:1905] T a35d94ca9bcc45cd92872dc8364f5520 P 23f9933d61c6400080f7e947702154bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:11:02.820457 19712 ts_tablet_manager.cc:1918] T a35d94ca9bcc45cd92872dc8364f5520 P c3ddbe3956a148f597f88da0aef3de4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:11:02.820647 19712 log.cc:1196] T a35d94ca9bcc45cd92872dc8364f5520 P c3ddbe3956a148f597f88da0aef3de4e: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-1/wal/wals/a35d94ca9bcc45cd92872dc8364f5520
I20260926 03:11:02.821149 19712 ts_tablet_manager.cc:1939] T a35d94ca9bcc45cd92872dc8364f5520 P c3ddbe3956a148f597f88da0aef3de4e: Deleting consensus metadata
I20260926 03:11:02.823154 19210 catalog_manager.cc:4841] TS c3ddbe3956a148f597f88da0aef3de4e (127.0.97.2:42011): tablet a35d94ca9bcc45cd92872dc8364f5520 (table test-table [id=92e316ad597a42a7bd22ff1755470a50]) successfully deleted
I20260926 03:11:02.824447 19711 ts_tablet_manager.cc:1918] T a35d94ca9bcc45cd92872dc8364f5520 P 23f9933d61c6400080f7e947702154bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:11:02.824581 19711 log.cc:1196] T a35d94ca9bcc45cd92872dc8364f5520 P 23f9933d61c6400080f7e947702154bb: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391950058021-388-0/minicluster-data/ts-2/wal/wals/a35d94ca9bcc45cd92872dc8364f5520
I20260926 03:11:02.824978 19711 ts_tablet_manager.cc:1939] T a35d94ca9bcc45cd92872dc8364f5520 P 23f9933d61c6400080f7e947702154bb: Deleting consensus metadata
I20260926 03:11:02.826529 19209 catalog_manager.cc:4841] TS 23f9933d61c6400080f7e947702154bb (127.0.97.3:44105): tablet a35d94ca9bcc45cd92872dc8364f5520 (table test-table [id=92e316ad597a42a7bd22ff1755470a50]) successfully deleted
I20260926 03:11:02.912058 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 19282
W20260926 03:11:02.926963 19207 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:40365 (error 108)
I20260926 03:11:02.927119 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 19413
I20260926 03:11:02.928907 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:02.928380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40365 (local address 127.0.0.1:45648)
0926 03:11:02.928577 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:02.928695 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 03:11:02.929168 19207 catalog_manager.cc:4584] TS 6eac465b436f4604b2adb68ca932ecfb (127.0.97.1:40365): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a35d94ca9bcc45cd92872dc8364f5520: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40365: connect: Connection refused (error 111)
I20260926 03:11:02.934473 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:02.927893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54260 (local address 127.0.97.2:42011)
0926 03:11:02.928051 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:02.928056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:02.928155 (+ 99us) server_negotiation.cc:407] Connection header received
0926 03:11:02.928359 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:02.928366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:02.928473 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:02.928580 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:02.929078 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.929891 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.931445 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:02.931854 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:02.932002 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:02.933050 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:02.933060 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:02.933074 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:02.933683 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:02.933738 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:02.933758 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:02.933868 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:02.934007 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 03:11:02.934109 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 03:11:02.935225 19535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:11:02.982944 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:02.982272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40365 (local address 127.0.0.1:45654)
0926 03:11:02.982557 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:02.982776 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260926 03:11:02.984006 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:02.983627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40365 (local address 127.0.0.1:45668)
0926 03:11:02.983777 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:02.983852 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 03:11:03.027019 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 19413
I20260926 03:11:03.041316 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 19543
I20260926 03:11:03.044304 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:03.043799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40365 (local address 127.0.0.1:45676)
0926 03:11:03.044005 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:03.044123 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 03:11:03.045333 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:03.044948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40365 (local address 127.0.0.1:45686)
0926 03:11:03.045096 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:03.045178 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 03:11:03.049409 19687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:03.042119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46078 (local address 127.0.97.3:44105)
0926 03:11:03.042310 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:03.042316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:03.042337 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:11:03.042588 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:03.042595 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:03.042706 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:03.042846 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:03.043249 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:03.044080 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:03.045880 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:03.046421 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:03.046601 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:03.047771 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:03.047782 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:03.047797 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:03.048426 (+ 629us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:03.048534 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:03.048559 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:03.048682 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:03.048907 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 03:11:03.049047 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:11:03.050112 19665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:11:03.126230 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:03.125707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40365 (local address 127.0.0.1:45698)
0926 03:11:03.125981 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:03.126072 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260926 03:11:03.127508 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:03.127017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40365 (local address 127.0.0.1:45706)
0926 03:11:03.127260 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:03.127353 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260926 03:11:03.144277 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 19543
I20260926 03:11:03.157630 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 19192
I20260926 03:11:03.158718 19253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:11:03.269111 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 19192
2026-09-26T03:11:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2732 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:11:03 dist-test-slave-hmbw krb5kdc[19722](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:11:03 dist-test-slave-hmbw krb5kdc[19722](info): set up 2 sockets
Sep 26 03:11:03 dist-test-slave-hmbw krb5kdc[19722](info): commencing operation
krb5kdc: starting...
W20260926 03:11:04.306545 388 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:11:04 dist-test-slave-hmbw krb5kdc[19722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:11:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:11:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:11:04.557953 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42243
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:11:04.891147 19735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:04.891467 19735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:04.891599 19735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:04.899094 19735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:11:04.899231 19735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:04.899312 19735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:04.899390 19735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:11:04.899453 19735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:11:04.909776 19735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:04.911197 19735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:04.912493 19735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:04.924150 19740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:04.924124 19743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:04.924831 19741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:04.925943 19735 server_base.cc:1034] running on GCE node
I20260926 03:11:04.926841 19735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:04.928411 19735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:04.929633 19735 hybrid_clock.cc:648] HybridClock initialized: now 1790392264929585 us; error 47 us; skew 500 ppm
I20260926 03:11:04.929986 19735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:11:04 dist-test-slave-hmbw krb5kdc[19722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:11:04.938611 19735 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:11:04.940312 19735 webserver.cc:460] Webserver started at http://127.0.97.62:40587/ using document root <none> and password file <none>
I20260926 03:11:04.941087 19735 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:04.941213 19735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:04.941519 19735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:04.944615 19735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "8ff9227778c54907972bc05c3da3401c"
format_stamp: "Formatted at 2026-09-26 03:11:04 on dist-test-slave-hmbw"
server_key: "42c65b0ef6a8dad82ba6955037365e44"
server_key_iv: "3fbdf809109aabd6ebd7eef5b235e522"
server_key_version: "encryptionkey@0"
I20260926 03:11:04.945390 19735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "8ff9227778c54907972bc05c3da3401c"
format_stamp: "Formatted at 2026-09-26 03:11:04 on dist-test-slave-hmbw"
server_key: "42c65b0ef6a8dad82ba6955037365e44"
server_key_iv: "3fbdf809109aabd6ebd7eef5b235e522"
server_key_version: "encryptionkey@0"
I20260926 03:11:04.950929 19735 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 03:11:04.954631 19750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:04.956259 19735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:11:04.956519 19735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "8ff9227778c54907972bc05c3da3401c"
format_stamp: "Formatted at 2026-09-26 03:11:04 on dist-test-slave-hmbw"
server_key: "42c65b0ef6a8dad82ba6955037365e44"
server_key_iv: "3fbdf809109aabd6ebd7eef5b235e522"
server_key_version: "encryptionkey@0"
I20260926 03:11:04.956724 19735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:04.987617 19735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:04.992208 19735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:04.992492 19735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:05.011099 19735 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42243
I20260926 03:11:05.011128 19801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42243 every 8 connection(s)
I20260926 03:11:05.012526 19735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:11:05.016425 19802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:05.022393 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 19735
I20260926 03:11:05.022625 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:11:05.023133 388 external_mini_cluster.cc:1428] Setting key 68ec7124dc82f0f2018cbf7a1d1c746e
I20260926 03:11:05.029455 19802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ff9227778c54907972bc05c3da3401c: Bootstrap starting.
I20260926 03:11:05.033370 19802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ff9227778c54907972bc05c3da3401c: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:05.034679 19802 log.cc:824] T 00000000000000000000000000000000 P 8ff9227778c54907972bc05c3da3401c: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:05.038141 19802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ff9227778c54907972bc05c3da3401c: No bootstrap required, opened a new log
Sep 26 03:11:05 dist-test-slave-hmbw krb5kdc[19722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:11:05.046631 19802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ff9227778c54907972bc05c3da3401c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff9227778c54907972bc05c3da3401c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42243 } }
I20260926 03:11:05.047083 19802 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ff9227778c54907972bc05c3da3401c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:05.047185 19802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ff9227778c54907972bc05c3da3401c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ff9227778c54907972bc05c3da3401c, State: Initialized, Role: FOLLOWER
I20260926 03:11:05.047765 19802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ff9227778c54907972bc05c3da3401c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff9227778c54907972bc05c3da3401c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42243 } }
I20260926 03:11:05.048015 19802 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ff9227778c54907972bc05c3da3401c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:11:05.048096 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:05.025153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33392 (local address 127.0.97.62:42243)
0926 03:11:05.026018 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:05.026034 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:05.026080 (+ 46us) server_negotiation.cc:407] Connection header received
0926 03:11:05.027988 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:05.028015 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:05.028475 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:05.028820 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:05.030676 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:05.032152 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:05.033985 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:05.034890 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:05.035041 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:05.039845 (+ 4804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:05.039854 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:05.039871 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:05.039925 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:05.043448 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:05.044227 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:05.044238 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:05.044248 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:05.044449 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:05.044850 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:05.044854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:05.044856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:05.045492 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:05.045789 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:05.046262 (+ 473us) server_negotiation.cc:299] Negotiation successful
0926 03:11:05.046763 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":362,"threads_started":1}
I20260926 03:11:05.048146 19802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ff9227778c54907972bc05c3da3401c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:11:05.048381 19802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ff9227778c54907972bc05c3da3401c [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:05.051244 19802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ff9227778c54907972bc05c3da3401c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff9227778c54907972bc05c3da3401c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42243 } }
I20260926 03:11:05.051774 19802 leader_election.cc:302] T 00000000000000000000000000000000 P 8ff9227778c54907972bc05c3da3401c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ff9227778c54907972bc05c3da3401c; no voters:
I20260926 03:11:05.052372 19802 leader_election.cc:288] T 00000000000000000000000000000000 P 8ff9227778c54907972bc05c3da3401c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:11:05.052548 19807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ff9227778c54907972bc05c3da3401c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:11:05.054387 19802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ff9227778c54907972bc05c3da3401c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:11:05.058858 19807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ff9227778c54907972bc05c3da3401c [term 1 LEADER]: Becoming Leader. State: Replica: 8ff9227778c54907972bc05c3da3401c, State: Running, Role: LEADER
I20260926 03:11:05.059441 19807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ff9227778c54907972bc05c3da3401c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff9227778c54907972bc05c3da3401c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42243 } }
I20260926 03:11:05.062108 19808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ff9227778c54907972bc05c3da3401c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ff9227778c54907972bc05c3da3401c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff9227778c54907972bc05c3da3401c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42243 } } }
I20260926 03:11:05.062363 19809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ff9227778c54907972bc05c3da3401c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ff9227778c54907972bc05c3da3401c. Latest consensus state: current_term: 1 leader_uuid: "8ff9227778c54907972bc05c3da3401c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff9227778c54907972bc05c3da3401c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42243 } } }
I20260926 03:11:05.062565 19808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ff9227778c54907972bc05c3da3401c [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:05.062608 19809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ff9227778c54907972bc05c3da3401c [sys.catalog]: This master's current role is: LEADER
W20260926 03:11:05.067269 19821 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8ff9227778c54907972bc05c3da3401c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:11:05.067389 19821 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:11:05.067541 19822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:11:05.069440 19822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:11:05.078374 19822 catalog_manager.cc:1347] Generated new cluster ID: 286a28e9ad564283aa1bcb871fe60492
I20260926 03:11:05.078485 19822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:11:05.093636 19822 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:11:05.095122 19822 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:11:05.103253 19822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ff9227778c54907972bc05c3da3401c: Generated new TSK 0
I20260926 03:11:05.104393 19822 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:11:05.170606 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42243
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:11:05.503145 19828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:05.503435 19828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:05.503530 19828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:05.510869 19828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:05.510993 19828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:05.511106 19828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:11:05.521108 19828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:05.522424 19828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:05.523783 19828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:05.537141 19834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:05.537278 19836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:05.537709 19833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:05.538012 19828 server_base.cc:1034] running on GCE node
I20260926 03:11:05.538740 19828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:05.539753 19828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:05.540992 19828 hybrid_clock.cc:648] HybridClock initialized: now 1790392265540937 us; error 52 us; skew 500 ppm
I20260926 03:11:05.541342 19828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:11:05 dist-test-slave-hmbw krb5kdc[19722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:11:05.544741 19828 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:11:05.546406 19828 webserver.cc:460] Webserver started at http://127.0.97.1:36333/ using document root <none> and password file <none>
I20260926 03:11:05.547144 19828 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:05.547271 19828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:05.547577 19828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:05.550724 19828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-0/data/instance:
uuid: "fa0d108fdf5e474186ea89619406ea93"
format_stamp: "Formatted at 2026-09-26 03:11:05 on dist-test-slave-hmbw"
server_key: "58c41217889d0b899e1086f08cbd156c"
server_key_iv: "8ab3cc9e3136f975e886f4baf544808d"
server_key_version: "encryptionkey@0"
I20260926 03:11:05.551535 19828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-0/wal/instance:
uuid: "fa0d108fdf5e474186ea89619406ea93"
format_stamp: "Formatted at 2026-09-26 03:11:05 on dist-test-slave-hmbw"
server_key: "58c41217889d0b899e1086f08cbd156c"
server_key_iv: "8ab3cc9e3136f975e886f4baf544808d"
server_key_version: "encryptionkey@0"
I20260926 03:11:05.557191 19828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 03:11:05.561247 19843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:05.562827 19828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:11:05.563014 19828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "fa0d108fdf5e474186ea89619406ea93"
format_stamp: "Formatted at 2026-09-26 03:11:05 on dist-test-slave-hmbw"
server_key: "58c41217889d0b899e1086f08cbd156c"
server_key_iv: "8ab3cc9e3136f975e886f4baf544808d"
server_key_version: "encryptionkey@0"
I20260926 03:11:05.563256 19828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:05.576117 19828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:05.580848 19828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:05.581138 19828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:05.582077 19828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:11:05.583420 19828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:11:05.583496 19828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:05.583618 19828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:11:05.583667 19828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:05.612387 19828 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44237
I20260926 03:11:05.612564 19955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44237 every 8 connection(s)
I20260926 03:11:05.613667 19828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:11:05.614138 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 19828
I20260926 03:11:05.614322 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:11:05.614972 388 external_mini_cluster.cc:1428] Setting key 72ee383da2b721a3b43aacdaa6973f46
Sep 26 03:11:05 dist-test-slave-hmbw krb5kdc[19722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:11:05.635155 19958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:05.616638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35155 (local address 127.0.97.62:42243)
0926 03:11:05.617158 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:05.617165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:05.618384 (+ 1219us) server_negotiation.cc:407] Connection header received
0926 03:11:05.620322 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:05.620328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:05.620414 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:05.620510 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:05.622154 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:05.622961 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:05.624625 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:05.625036 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:05.625149 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:05.629307 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:05.629313 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:05.629317 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:05.629409 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:05.631946 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:05.632739 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:05.632744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:05.632746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:05.632874 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:05.633310 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:05.633315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:05.633317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:05.633600 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:05.633749 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:05.634545 (+ 796us) server_negotiation.cc:299] Negotiation successful
0926 03:11:05.634796 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":280,"threads_started":1}
I20260926 03:11:05.636197 19959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:05.617309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42243 (local address 127.0.97.1:35155)
0926 03:11:05.618148 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:05.618189 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:05.620031 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:05.620777 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:05.620786 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:05.621337 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:11:05.621968 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:05.621986 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:05.623125 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:05.623129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:05.624446 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:05.624466 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:05.625287 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:05.625296 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:05.625524 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:05.626206 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:11:05.626233 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:11:05.629008 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:11:05.632116 (+ 3108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:05.632127 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:05.632145 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:05.632611 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:05.632987 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:05.632993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:05.632996 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:05.633192 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:05.633731 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:11:05.633740 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:11:05.634118 (+ 378us) client_negotiation.cc:769] Sending connection context
0926 03:11:05.634393 (+ 275us) client_negotiation.cc:240] Negotiation successful
0926 03:11:05.635007 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":366,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:11:05.637837 19956 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42243
I20260926 03:11:05.638134 19956 heartbeater.cc:464] Registering TS with master...
I20260926 03:11:05.638935 19956 heartbeater.cc:511] Master 127.0.97.62:42243 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 03:11:05.641251 19767 ts_manager.cc:194] Registered new tserver with Master: fa0d108fdf5e474186ea89619406ea93 (127.0.97.1:44237)
I20260926 03:11:05.642804 19767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:11:05.669528 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42243
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:11:06.003487 19962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:06.003820 19962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:06.003963 19962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:06.011548 19962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:06.011664 19962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:06.011812 19962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:11:06.022569 19962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:06.024271 19962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:06.026139 19962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:06.040019 19970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:06.040077 19967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:06.040043 19968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:06.040650 19962 server_base.cc:1034] running on GCE node
I20260926 03:11:06.041380 19962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:06.042418 19962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:06.043658 19962 hybrid_clock.cc:648] HybridClock initialized: now 1790392266043607 us; error 44 us; skew 500 ppm
I20260926 03:11:06.044001 19962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:11:06 dist-test-slave-hmbw krb5kdc[19722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:11:06.047401 19962 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:11:06.049124 19962 webserver.cc:460] Webserver started at http://127.0.97.2:43545/ using document root <none> and password file <none>
I20260926 03:11:06.049839 19962 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:06.049943 19962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:06.050211 19962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:06.053375 19962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-1/data/instance:
uuid: "f6b3cefe617e4895b864ec10fe921174"
format_stamp: "Formatted at 2026-09-26 03:11:06 on dist-test-slave-hmbw"
server_key: "61ba3ccba4743ea8983c648d8c1b758c"
server_key_iv: "114f5584922afaa5c078249b5a2344ae"
server_key_version: "encryptionkey@0"
I20260926 03:11:06.054162 19962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f6b3cefe617e4895b864ec10fe921174"
format_stamp: "Formatted at 2026-09-26 03:11:06 on dist-test-slave-hmbw"
server_key: "61ba3ccba4743ea8983c648d8c1b758c"
server_key_iv: "114f5584922afaa5c078249b5a2344ae"
server_key_version: "encryptionkey@0"
I20260926 03:11:06.059947 19962 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260926 03:11:06.064028 19977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:06.065459 19962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:11:06.065616 19962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "f6b3cefe617e4895b864ec10fe921174"
format_stamp: "Formatted at 2026-09-26 03:11:06 on dist-test-slave-hmbw"
server_key: "61ba3ccba4743ea8983c648d8c1b758c"
server_key_iv: "114f5584922afaa5c078249b5a2344ae"
server_key_version: "encryptionkey@0"
I20260926 03:11:06.065830 19962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:06.114419 19962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:06.119231 19962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:06.119532 19962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:06.120570 19962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:11:06.122607 19962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:11:06.122702 19962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:06.122840 19962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:11:06.122907 19962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:06.157330 19962 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39169
I20260926 03:11:06.157341 20089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39169 every 8 connection(s)
I20260926 03:11:06.158674 19962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:11:06.163813 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 19962
I20260926 03:11:06.164002 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:11:06.164441 388 external_mini_cluster.cc:1428] Setting key 4b9016e18e5e1482b2164ea7a6315fa6
Sep 26 03:11:06 dist-test-slave-hmbw krb5kdc[19722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:11:06.180212 20092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:06.161136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40649 (local address 127.0.97.62:42243)
0926 03:11:06.161600 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:06.161607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:06.162702 (+ 1095us) server_negotiation.cc:407] Connection header received
0926 03:11:06.164758 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:06.164764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:06.164862 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:06.164964 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:06.166713 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.167548 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.169277 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.169796 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.169912 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:06.173842 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:06.173848 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:06.173851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:06.173889 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:06.176739 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:06.177660 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:06.177667 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:06.177671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:06.177828 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:06.178317 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:06.178321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:06.178323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:06.178673 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:06.178826 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:06.179587 (+ 761us) server_negotiation.cc:299] Negotiation successful
0926 03:11:06.179857 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":229,"threads_started":1}
I20260926 03:11:06.181236 20093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:06.161684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42243 (local address 127.0.97.2:40649)
0926 03:11:06.162491 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:06.162581 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:06.164422 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:06.165269 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:06.165278 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:06.165852 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:11:06.166484 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:06.166543 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.167734 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.167739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:06.169096 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:06.169111 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.170065 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.170072 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:06.170313 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:06.171106 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:11:06.171136 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:11:06.173539 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:11:06.176939 (+ 3400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:06.176949 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:06.176965 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:06.177498 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:06.177951 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:06.177956 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:06.177960 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:06.178185 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:06.178808 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:11:06.178816 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:11:06.179199 (+ 383us) client_negotiation.cc:769] Sending connection context
0926 03:11:06.179461 (+ 262us) client_negotiation.cc:240] Negotiation successful
0926 03:11:06.180024 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":298,"threads_started":1}
I20260926 03:11:06.182926 20090 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42243
I20260926 03:11:06.183276 20090 heartbeater.cc:464] Registering TS with master...
I20260926 03:11:06.184222 20090 heartbeater.cc:511] Master 127.0.97.62:42243 requested a full tablet report, sending...
I20260926 03:11:06.186182 19767 ts_manager.cc:194] Registered new tserver with Master: f6b3cefe617e4895b864ec10fe921174 (127.0.97.2:39169)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 03:11:06.187340 19767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:11:06.217845 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42243
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:11:06.547861 20096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:06.548146 20096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:06.548251 20096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:06.555797 20096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:06.555926 20096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:06.556043 20096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:11:06.566262 20096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:06.567678 20096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:06.569010 20096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:06.582321 20101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:06.582398 20102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:06.582369 20104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:06.583168 20096 server_base.cc:1034] running on GCE node
I20260926 03:11:06.583755 20096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:06.584779 20096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:06.586015 20096 hybrid_clock.cc:648] HybridClock initialized: now 1790392266585950 us; error 55 us; skew 500 ppm
I20260926 03:11:06.586392 20096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:11:06 dist-test-slave-hmbw krb5kdc[19722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:11:06.589602 20096 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:11:06.591152 20096 webserver.cc:460] Webserver started at http://127.0.97.3:38093/ using document root <none> and password file <none>
I20260926 03:11:06.591842 20096 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:06.591950 20096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:06.592253 20096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:06.595317 20096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-2/data/instance:
uuid: "e6270617a04045a4b47ab30e53ec1410"
format_stamp: "Formatted at 2026-09-26 03:11:06 on dist-test-slave-hmbw"
server_key: "fb5589f0792298eaed9a0af39c036eea"
server_key_iv: "240dc0b915e87ecea96b2039bcdda22d"
server_key_version: "encryptionkey@0"
I20260926 03:11:06.596120 20096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e6270617a04045a4b47ab30e53ec1410"
format_stamp: "Formatted at 2026-09-26 03:11:06 on dist-test-slave-hmbw"
server_key: "fb5589f0792298eaed9a0af39c036eea"
server_key_iv: "240dc0b915e87ecea96b2039bcdda22d"
server_key_version: "encryptionkey@0"
I20260926 03:11:06.601586 20096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:11:06.605496 20111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:06.607059 20096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:11:06.607226 20096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "e6270617a04045a4b47ab30e53ec1410"
format_stamp: "Formatted at 2026-09-26 03:11:06 on dist-test-slave-hmbw"
server_key: "fb5589f0792298eaed9a0af39c036eea"
server_key_iv: "240dc0b915e87ecea96b2039bcdda22d"
server_key_version: "encryptionkey@0"
I20260926 03:11:06.607451 20096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:06.646099 19956 heartbeater.cc:503] Master 127.0.97.62:42243 was elected leader, sending a full tablet report...
I20260926 03:11:06.648794 20096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:06.653365 20096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:06.653671 20096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:06.654871 20096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:11:06.656397 20096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:11:06.656493 20096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:06.656613 20096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:11:06.656664 20096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:06.688922 20096 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38181
I20260926 03:11:06.688994 20223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38181 every 8 connection(s)
I20260926 03:11:06.690191 20096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:11:06.692524 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 20096
I20260926 03:11:06.692691 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:11:06.693276 388 external_mini_cluster.cc:1428] Setting key d17fa3da5308b2c0c7b020d9b62944c0
Sep 26 03:11:06 dist-test-slave-hmbw krb5kdc[19722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:11:06.712666 20226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:06.692939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36995 (local address 127.0.97.62:42243)
0926 03:11:06.693546 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:06.693553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:06.694463 (+ 910us) server_negotiation.cc:407] Connection header received
0926 03:11:06.696693 (+ 2230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:06.696698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:06.696781 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:06.696879 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:06.698613 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.699415 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.701512 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.701994 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.702111 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:06.706232 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:06.706238 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:06.706242 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:06.706286 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:06.709133 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:06.710051 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:06.710056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:06.710058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:06.710184 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:06.710696 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:06.710700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:06.710703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:06.711023 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:06.711164 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:06.712027 (+ 863us) server_negotiation.cc:299] Negotiation successful
0926 03:11:06.712289 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":240,"threads_started":1}
I20260926 03:11:06.713692 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:06.693439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42243 (local address 127.0.97.3:36995)
0926 03:11:06.694295 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:06.694349 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:06.696414 (+ 2065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:06.697155 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:06.697164 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:06.697761 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:11:06.698406 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:06.698425 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.699567 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.699571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:06.701311 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:06.701329 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.702461 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.702468 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:06.702788 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:06.703537 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:11:06.703563 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:11:06.705954 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:11:06.709314 (+ 3360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:06.709322 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:06.709339 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:06.709891 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:06.710290 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:06.710293 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:06.710296 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:06.710593 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:06.711165 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:11:06.711174 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:11:06.711555 (+ 381us) client_negotiation.cc:769] Sending connection context
0926 03:11:06.711852 (+ 297us) client_negotiation.cc:240] Negotiation successful
0926 03:11:06.712477 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":351,"threads_started":1}
I20260926 03:11:06.715222 20224 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42243
I20260926 03:11:06.715631 20224 heartbeater.cc:464] Registering TS with master...
I20260926 03:11:06.716392 20224 heartbeater.cc:511] Master 127.0.97.62:42243 requested a full tablet report, sending...
I20260926 03:11:06.718063 19767 ts_manager.cc:194] Registered new tserver with Master: e6270617a04045a4b47ab30e53ec1410 (127.0.97.3:38181)
I20260926 03:11:06.719028 19767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36995
I20260926 03:11:06.721725 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:11:06.739590 20226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:06.726633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33398 (local address 127.0.97.62:42243)
0926 03:11:06.726885 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:06.726892 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:06.727256 (+ 364us) server_negotiation.cc:407] Connection header received
0926 03:11:06.727451 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:06.727457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:06.727551 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:06.727644 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:06.729030 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.729832 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.731216 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.731663 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.731783 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:06.734188 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:06.734194 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:06.734197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:06.734249 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:06.736833 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:06.737522 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:06.737528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:06.737532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:06.737661 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:06.738009 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:06.738013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:06.738015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:06.738340 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:06.738612 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:06.739038 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 03:11:06.739269 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 03:11:06.743713 19767 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:11:06.750828 19767 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:11:06.776633 20239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:06.764069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56842 (local address 127.0.97.3:38181)
0926 03:11:06.764814 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:06.764821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:06.766203 (+ 1382us) server_negotiation.cc:407] Connection header received
0926 03:11:06.766472 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:06.766482 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:06.766776 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:06.766983 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:11:06.767878 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.769860 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.773713 (+ 3853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.775278 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.775399 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:06.775637 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:06.776028 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 03:11:06.776218 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":430,"threads_started":1}
I20260926 03:11:06.777560 20238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:06.763819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38181 (local address 127.0.0.1:56842)
0926 03:11:06.766045 (+ 2226us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:06.766078 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:06.766263 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:06.767016 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:06.767025 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:06.767078 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:11:06.767542 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:06.767559 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.770195 (+ 2636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.770202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:06.773468 (+ 3266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:06.773488 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.775484 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.775490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:06.775726 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:06.775787 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 03:11:06.776893 (+ 1106us) client_negotiation.cc:240] Negotiation successful
0926 03:11:06.777131 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2003,"mutex_wait_us":77,"spinlock_wait_cycles":1920,"thread_start_us":2782,"threads_started":1}
I20260926 03:11:06.778227 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:06.764043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44237 (local address 127.0.0.1:54132)
0926 03:11:06.766852 (+ 2809us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:06.766871 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:06.767058 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:06.767883 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:06.767888 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:06.767914 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:11:06.768390 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:06.768413 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.770729 (+ 2316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.770735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:06.774851 (+ 4116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:06.774868 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.777460 (+ 2592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.777465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:06.777593 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:06.777616 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 03:11:06.777698 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 03:11:06.777830 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2692,"mutex_wait_us":1749,"thread_start_us":1690,"threads_started":1}
I20260926 03:11:06.778644 20240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:06.764511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54132 (local address 127.0.97.1:44237)
0926 03:11:06.765604 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:06.765611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:06.766959 (+ 1348us) server_negotiation.cc:407] Connection header received
0926 03:11:06.767235 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:06.767243 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:06.767451 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:06.767596 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:11:06.768607 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.770522 (+ 1915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.775059 (+ 4537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.776779 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.777069 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:06.777285 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:06.778093 (+ 808us) server_negotiation.cc:299] Negotiation successful
0926 03:11:06.778268 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":924,"thread_start_us":756,"threads_started":1}
I20260926 03:11:06.781368 20159 tablet_service.cc:1467] Processing CreateTablet for tablet c5ac7c3031e24643ba3d0fe0e1a03a44 (DEFAULT_TABLE table=test-table [id=5274cab19af649f096bbc452b52ce6f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:11:06.782004 19891 tablet_service.cc:1467] Processing CreateTablet for tablet c5ac7c3031e24643ba3d0fe0e1a03a44 (DEFAULT_TABLE table=test-table [id=5274cab19af649f096bbc452b52ce6f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:11:06.783033 20159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5ac7c3031e24643ba3d0fe0e1a03a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:06.783461 19891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5ac7c3031e24643ba3d0fe0e1a03a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:06.783828 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:06.767052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39169 (local address 127.0.0.1:44756)
0926 03:11:06.774060 (+ 7008us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:06.774081 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:06.774200 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:06.774918 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:06.774924 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:06.774951 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:11:06.775259 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:06.775268 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.778052 (+ 2784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.778057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:06.780991 (+ 2934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:06.781011 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.782931 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.782939 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:06.783161 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:06.783198 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 03:11:06.783294 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 03:11:06.783440 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6882,"thread_start_us":6726,"threads_started":1}
I20260926 03:11:06.784097 20242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:06.767568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44756 (local address 127.0.97.2:39169)
0926 03:11:06.774349 (+ 6781us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:06.774356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:06.774381 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:11:06.774464 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:06.774471 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:06.774727 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:06.774916 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:11:06.775402 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.777876 (+ 2474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.781227 (+ 3351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.782716 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.782861 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:06.783069 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:06.783364 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 03:11:06.783541 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6592,"thread_start_us":6492,"threads_started":1}
I20260926 03:11:06.788271 20025 tablet_service.cc:1467] Processing CreateTablet for tablet c5ac7c3031e24643ba3d0fe0e1a03a44 (DEFAULT_TABLE table=test-table [id=5274cab19af649f096bbc452b52ce6f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:11:06.789896 20025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5ac7c3031e24643ba3d0fe0e1a03a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:06.797189 20243 tablet_bootstrap.cc:492] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93: Bootstrap starting.
I20260926 03:11:06.798408 20244 tablet_bootstrap.cc:492] T c5ac7c3031e24643ba3d0fe0e1a03a44 P e6270617a04045a4b47ab30e53ec1410: Bootstrap starting.
I20260926 03:11:06.802205 20243 tablet_bootstrap.cc:654] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:06.802205 20244 tablet_bootstrap.cc:654] T c5ac7c3031e24643ba3d0fe0e1a03a44 P e6270617a04045a4b47ab30e53ec1410: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:06.803598 20244 log.cc:824] T c5ac7c3031e24643ba3d0fe0e1a03a44 P e6270617a04045a4b47ab30e53ec1410: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:06.803599 20243 log.cc:824] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:06.804199 20245 tablet_bootstrap.cc:492] T c5ac7c3031e24643ba3d0fe0e1a03a44 P f6b3cefe617e4895b864ec10fe921174: Bootstrap starting.
I20260926 03:11:06.807777 20245 tablet_bootstrap.cc:654] T c5ac7c3031e24643ba3d0fe0e1a03a44 P f6b3cefe617e4895b864ec10fe921174: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:06.809182 20244 tablet_bootstrap.cc:492] T c5ac7c3031e24643ba3d0fe0e1a03a44 P e6270617a04045a4b47ab30e53ec1410: No bootstrap required, opened a new log
I20260926 03:11:06.809130 20245 log.cc:824] T c5ac7c3031e24643ba3d0fe0e1a03a44 P f6b3cefe617e4895b864ec10fe921174: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:06.809398 20243 tablet_bootstrap.cc:492] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93: No bootstrap required, opened a new log
I20260926 03:11:06.809468 20244 ts_tablet_manager.cc:1397] T c5ac7c3031e24643ba3d0fe0e1a03a44 P e6270617a04045a4b47ab30e53ec1410: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260926 03:11:06.809684 20243 ts_tablet_manager.cc:1397] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.002s
I20260926 03:11:06.812548 20245 tablet_bootstrap.cc:492] T c5ac7c3031e24643ba3d0fe0e1a03a44 P f6b3cefe617e4895b864ec10fe921174: No bootstrap required, opened a new log
I20260926 03:11:06.812834 20245 ts_tablet_manager.cc:1397] T c5ac7c3031e24643ba3d0fe0e1a03a44 P f6b3cefe617e4895b864ec10fe921174: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 03:11:06.817900 20244 raft_consensus.cc:348] T c5ac7c3031e24643ba3d0fe0e1a03a44 P e6270617a04045a4b47ab30e53ec1410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6270617a04045a4b47ab30e53ec1410" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38181 } } peers { permanent_uuid: "fa0d108fdf5e474186ea89619406ea93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44237 } } peers { permanent_uuid: "f6b3cefe617e4895b864ec10fe921174" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39169 } }
I20260926 03:11:06.818334 20244 raft_consensus.cc:374] T c5ac7c3031e24643ba3d0fe0e1a03a44 P e6270617a04045a4b47ab30e53ec1410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:06.818434 20244 raft_consensus.cc:729] T c5ac7c3031e24643ba3d0fe0e1a03a44 P e6270617a04045a4b47ab30e53ec1410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6270617a04045a4b47ab30e53ec1410, State: Initialized, Role: FOLLOWER
I20260926 03:11:06.819110 20244 consensus_queue.cc:260] T c5ac7c3031e24643ba3d0fe0e1a03a44 P e6270617a04045a4b47ab30e53ec1410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6270617a04045a4b47ab30e53ec1410" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38181 } } peers { permanent_uuid: "fa0d108fdf5e474186ea89619406ea93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44237 } } peers { permanent_uuid: "f6b3cefe617e4895b864ec10fe921174" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39169 } }
I20260926 03:11:06.820546 20224 heartbeater.cc:503] Master 127.0.97.62:42243 was elected leader, sending a full tablet report...
I20260926 03:11:06.820412 20243 raft_consensus.cc:348] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6270617a04045a4b47ab30e53ec1410" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38181 } } peers { permanent_uuid: "fa0d108fdf5e474186ea89619406ea93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44237 } } peers { permanent_uuid: "f6b3cefe617e4895b864ec10fe921174" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39169 } }
I20260926 03:11:06.820870 20243 raft_consensus.cc:374] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:06.820967 20243 raft_consensus.cc:729] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa0d108fdf5e474186ea89619406ea93, State: Initialized, Role: FOLLOWER
I20260926 03:11:06.821233 20244 ts_tablet_manager.cc:1428] T c5ac7c3031e24643ba3d0fe0e1a03a44 P e6270617a04045a4b47ab30e53ec1410: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260926 03:11:06.821702 20243 consensus_queue.cc:260] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6270617a04045a4b47ab30e53ec1410" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38181 } } peers { permanent_uuid: "fa0d108fdf5e474186ea89619406ea93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44237 } } peers { permanent_uuid: "f6b3cefe617e4895b864ec10fe921174" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39169 } }
I20260926 03:11:06.824079 20243 ts_tablet_manager.cc:1428] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93: Time spent starting tablet: real 0.014s user 0.007s sys 0.009s
I20260926 03:11:06.823833 20245 raft_consensus.cc:348] T c5ac7c3031e24643ba3d0fe0e1a03a44 P f6b3cefe617e4895b864ec10fe921174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6270617a04045a4b47ab30e53ec1410" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38181 } } peers { permanent_uuid: "fa0d108fdf5e474186ea89619406ea93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44237 } } peers { permanent_uuid: "f6b3cefe617e4895b864ec10fe921174" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39169 } }
I20260926 03:11:06.824393 20245 raft_consensus.cc:374] T c5ac7c3031e24643ba3d0fe0e1a03a44 P f6b3cefe617e4895b864ec10fe921174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:06.824489 20245 raft_consensus.cc:729] T c5ac7c3031e24643ba3d0fe0e1a03a44 P f6b3cefe617e4895b864ec10fe921174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b3cefe617e4895b864ec10fe921174, State: Initialized, Role: FOLLOWER
I20260926 03:11:06.825258 20245 consensus_queue.cc:260] T c5ac7c3031e24643ba3d0fe0e1a03a44 P f6b3cefe617e4895b864ec10fe921174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6270617a04045a4b47ab30e53ec1410" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38181 } } peers { permanent_uuid: "fa0d108fdf5e474186ea89619406ea93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44237 } } peers { permanent_uuid: "f6b3cefe617e4895b864ec10fe921174" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39169 } }
I20260926 03:11:06.826900 20090 heartbeater.cc:503] Master 127.0.97.62:42243 was elected leader, sending a full tablet report...
I20260926 03:11:06.827683 20245 ts_tablet_manager.cc:1428] T c5ac7c3031e24643ba3d0fe0e1a03a44 P f6b3cefe617e4895b864ec10fe921174: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
W20260926 03:11:06.866439 19957 tablet.cc:2369] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:11:06.886219 20250 raft_consensus.cc:482] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:11:06.886644 20250 raft_consensus.cc:504] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6270617a04045a4b47ab30e53ec1410" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38181 } } peers { permanent_uuid: "fa0d108fdf5e474186ea89619406ea93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44237 } } peers { permanent_uuid: "f6b3cefe617e4895b864ec10fe921174" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39169 } }
I20260926 03:11:06.888365 20250 leader_election.cc:288] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6270617a04045a4b47ab30e53ec1410 (127.0.97.3:38181), f6b3cefe617e4895b864ec10fe921174 (127.0.97.2:39169)
I20260926 03:11:06.895362 20239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:06.888872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40517 (local address 127.0.97.3:38181)
0926 03:11:06.889035 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:06.889040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:06.889353 (+ 313us) server_negotiation.cc:407] Connection header received
0926 03:11:06.889573 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:06.889580 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:06.889660 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:06.889719 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:11:06.890485 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.891451 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.893432 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.894382 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.894481 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:06.894601 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:06.894976 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 03:11:06.895113 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 03:11:06.895506 20252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:06.888598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38181 (local address 127.0.97.1:40517)
0926 03:11:06.889241 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:06.889257 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:06.889404 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:06.889936 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:06.889940 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:06.889970 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:11:06.890334 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:06.890346 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.891587 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.891592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:06.893253 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:06.893266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.894618 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.894624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:06.894788 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:06.894824 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 03:11:06.894919 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 03:11:06.895062 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":47,"thread_start_us":440,"threads_started":1}
I20260926 03:11:06.895705 20242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:06.889867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58553 (local address 127.0.97.2:39169)
0926 03:11:06.890058 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:06.890064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:06.890165 (+ 101us) server_negotiation.cc:407] Connection header received
0926 03:11:06.890338 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:06.890343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:06.890431 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:06.890544 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:11:06.891050 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.891880 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.893808 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.894754 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.894884 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:06.894975 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:06.895335 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 03:11:06.895470 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:11:06.896245 20253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:06.889448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39169 (local address 127.0.97.1:58553)
0926 03:11:06.890041 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:06.890062 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:06.890205 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:06.890558 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:06.890562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:06.890589 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:11:06.890918 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:06.890929 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.892012 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.892015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:06.893652 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:06.893666 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.894980 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.894986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:06.895172 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:06.895204 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 03:11:06.895390 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 03:11:06.895968 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":36,"thread_start_us":614,"threads_started":1}
I20260926 03:11:06.896534 20179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ac7c3031e24643ba3d0fe0e1a03a44" candidate_uuid: "fa0d108fdf5e474186ea89619406ea93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6270617a04045a4b47ab30e53ec1410" is_pre_election: true
I20260926 03:11:06.897017 20179 raft_consensus.cc:2454] T c5ac7c3031e24643ba3d0fe0e1a03a44 P e6270617a04045a4b47ab30e53ec1410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa0d108fdf5e474186ea89619406ea93 in term 0.
I20260926 03:11:06.897192 20045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ac7c3031e24643ba3d0fe0e1a03a44" candidate_uuid: "fa0d108fdf5e474186ea89619406ea93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3cefe617e4895b864ec10fe921174" is_pre_election: true
I20260926 03:11:06.897663 20045 raft_consensus.cc:2454] T c5ac7c3031e24643ba3d0fe0e1a03a44 P f6b3cefe617e4895b864ec10fe921174 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa0d108fdf5e474186ea89619406ea93 in term 0.
I20260926 03:11:06.897671 19845 leader_election.cc:302] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6270617a04045a4b47ab30e53ec1410, fa0d108fdf5e474186ea89619406ea93; no voters:
I20260926 03:11:06.898132 20250 raft_consensus.cc:2781] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:11:06.898274 20250 raft_consensus.cc:482] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:11:06.898373 20250 raft_consensus.cc:3037] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:06.900660 20250 raft_consensus.cc:504] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6270617a04045a4b47ab30e53ec1410" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38181 } } peers { permanent_uuid: "fa0d108fdf5e474186ea89619406ea93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44237 } } peers { permanent_uuid: "f6b3cefe617e4895b864ec10fe921174" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39169 } }
I20260926 03:11:06.901382 20250 leader_election.cc:288] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93 [CANDIDATE]: Term 1 election: Requested vote from peers e6270617a04045a4b47ab30e53ec1410 (127.0.97.3:38181), f6b3cefe617e4895b864ec10fe921174 (127.0.97.2:39169)
I20260926 03:11:06.901984 20045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ac7c3031e24643ba3d0fe0e1a03a44" candidate_uuid: "fa0d108fdf5e474186ea89619406ea93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b3cefe617e4895b864ec10fe921174"
I20260926 03:11:06.901984 20179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ac7c3031e24643ba3d0fe0e1a03a44" candidate_uuid: "fa0d108fdf5e474186ea89619406ea93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6270617a04045a4b47ab30e53ec1410"
I20260926 03:11:06.902217 20179 raft_consensus.cc:3037] T c5ac7c3031e24643ba3d0fe0e1a03a44 P e6270617a04045a4b47ab30e53ec1410 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:06.902217 20045 raft_consensus.cc:3037] T c5ac7c3031e24643ba3d0fe0e1a03a44 P f6b3cefe617e4895b864ec10fe921174 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:06.905539 20179 raft_consensus.cc:2454] T c5ac7c3031e24643ba3d0fe0e1a03a44 P e6270617a04045a4b47ab30e53ec1410 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa0d108fdf5e474186ea89619406ea93 in term 1.
I20260926 03:11:06.905539 20045 raft_consensus.cc:2454] T c5ac7c3031e24643ba3d0fe0e1a03a44 P f6b3cefe617e4895b864ec10fe921174 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa0d108fdf5e474186ea89619406ea93 in term 1.
I20260926 03:11:06.906102 19845 leader_election.cc:302] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6b3cefe617e4895b864ec10fe921174, fa0d108fdf5e474186ea89619406ea93; no voters:
I20260926 03:11:06.906446 20250 raft_consensus.cc:2781] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:11:06.907025 20250 raft_consensus.cc:686] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93 [term 1 LEADER]: Becoming Leader. State: Replica: fa0d108fdf5e474186ea89619406ea93, State: Running, Role: LEADER
I20260926 03:11:06.907649 20250 consensus_queue.cc:237] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6270617a04045a4b47ab30e53ec1410" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38181 } } peers { permanent_uuid: "fa0d108fdf5e474186ea89619406ea93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44237 } } peers { permanent_uuid: "f6b3cefe617e4895b864ec10fe921174" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39169 } }
W20260926 03:11:06.910709 20091 tablet.cc:2369] T c5ac7c3031e24643ba3d0fe0e1a03a44 P f6b3cefe617e4895b864ec10fe921174: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:11:06.913178 19767 catalog_manager.cc:5495] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa0d108fdf5e474186ea89619406ea93 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "fa0d108fdf5e474186ea89619406ea93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6270617a04045a4b47ab30e53ec1410" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa0d108fdf5e474186ea89619406ea93" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6b3cefe617e4895b864ec10fe921174" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39169 } health_report { overall_health: UNKNOWN } } }
I20260926 03:11:06.932644 20240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:06.926938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54134 (local address 127.0.97.1:44237)
0926 03:11:06.927164 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:06.927171 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:06.927244 (+ 73us) server_negotiation.cc:407] Connection header received
0926 03:11:06.927497 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:06.927501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:06.927585 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:06.927725 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:11:06.928144 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.928900 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.930304 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.930762 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.930879 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:06.931412 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:11:06.931942 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:11:06.932083 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:06.932188 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 03:11:06.932277 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 03:11:06.941581 20045 raft_consensus.cc:1261] T c5ac7c3031e24643ba3d0fe0e1a03a44 P f6b3cefe617e4895b864ec10fe921174 [term 1 FOLLOWER]: Refusing update from remote peer fa0d108fdf5e474186ea89619406ea93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:11:06.941766 20179 raft_consensus.cc:1261] T c5ac7c3031e24643ba3d0fe0e1a03a44 P e6270617a04045a4b47ab30e53ec1410 [term 1 FOLLOWER]: Refusing update from remote peer fa0d108fdf5e474186ea89619406ea93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260926 03:11:06.941811 20225 tablet.cc:2369] T c5ac7c3031e24643ba3d0fe0e1a03a44 P e6270617a04045a4b47ab30e53ec1410: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:11:06.942857 20254 consensus_queue.cc:1035] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6b3cefe617e4895b864ec10fe921174" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:11:06.943240 20250 consensus_queue.cc:1035] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6270617a04045a4b47ab30e53ec1410" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:11:06.955245 20257 mvcc.cc:205] Tried to move back new op lower bound from 7333446725382086656 to 7333446725257953280. Current Snapshot: MvccSnapshot[applied={T|T < 7333446725382086656}]
I20260926 03:11:06.968806 19766 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:11:06.969168 19766 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:11:06.973500 19766 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ff9227778c54907972bc05c3da3401c: Sending DeleteTablet for 3 replicas of tablet c5ac7c3031e24643ba3d0fe0e1a03a44
I20260926 03:11:06.974881 19891 tablet_service.cc:1514] Processing DeleteTablet for tablet c5ac7c3031e24643ba3d0fe0e1a03a44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:11:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54132
I20260926 03:11:06.974897 20159 tablet_service.cc:1514] Processing DeleteTablet for tablet c5ac7c3031e24643ba3d0fe0e1a03a44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:11:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56842
I20260926 03:11:06.975674 20025 tablet_service.cc:1514] Processing DeleteTablet for tablet c5ac7c3031e24643ba3d0fe0e1a03a44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:11:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44756
I20260926 03:11:06.976238 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 19828
I20260926 03:11:06.988503 20267 tablet_replica.cc:330] T c5ac7c3031e24643ba3d0fe0e1a03a44 P e6270617a04045a4b47ab30e53ec1410: stopping tablet replica
I20260926 03:11:06.988775 20269 tablet_replica.cc:330] T c5ac7c3031e24643ba3d0fe0e1a03a44 P f6b3cefe617e4895b864ec10fe921174: stopping tablet replica
I20260926 03:11:06.989044 20267 raft_consensus.cc:2229] T c5ac7c3031e24643ba3d0fe0e1a03a44 P e6270617a04045a4b47ab30e53ec1410 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:11:06.989195 20269 raft_consensus.cc:2229] T c5ac7c3031e24643ba3d0fe0e1a03a44 P f6b3cefe617e4895b864ec10fe921174 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:11:06.989590 20267 raft_consensus.cc:2258] T c5ac7c3031e24643ba3d0fe0e1a03a44 P e6270617a04045a4b47ab30e53ec1410 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:11:06.989665 20269 raft_consensus.cc:2258] T c5ac7c3031e24643ba3d0fe0e1a03a44 P f6b3cefe617e4895b864ec10fe921174 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:11:06.990650 20268 tablet_replica.cc:330] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93: stopping tablet replica
I20260926 03:11:06.991134 20268 raft_consensus.cc:2229] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:11:06.991879 20268 raft_consensus.cc:2258] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:11:06.993336 20269 ts_tablet_manager.cc:1905] T c5ac7c3031e24643ba3d0fe0e1a03a44 P f6b3cefe617e4895b864ec10fe921174: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:11:06.993335 20267 ts_tablet_manager.cc:1905] T c5ac7c3031e24643ba3d0fe0e1a03a44 P e6270617a04045a4b47ab30e53ec1410: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 03:11:06 dist-test-slave-hmbw krb5kdc[19722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:11:07.000536 20268 ts_tablet_manager.cc:1905] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:11:07.000682 20269 ts_tablet_manager.cc:1918] T c5ac7c3031e24643ba3d0fe0e1a03a44 P f6b3cefe617e4895b864ec10fe921174: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:11:07.000844 20269 log.cc:1196] T c5ac7c3031e24643ba3d0fe0e1a03a44 P f6b3cefe617e4895b864ec10fe921174: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-1/wal/wals/c5ac7c3031e24643ba3d0fe0e1a03a44
I20260926 03:11:07.001215 20267 ts_tablet_manager.cc:1918] T c5ac7c3031e24643ba3d0fe0e1a03a44 P e6270617a04045a4b47ab30e53ec1410: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:11:07.001367 20269 ts_tablet_manager.cc:1939] T c5ac7c3031e24643ba3d0fe0e1a03a44 P f6b3cefe617e4895b864ec10fe921174: Deleting consensus metadata
I20260926 03:11:07.001624 20267 log.cc:1196] T c5ac7c3031e24643ba3d0fe0e1a03a44 P e6270617a04045a4b47ab30e53ec1410: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-2/wal/wals/c5ac7c3031e24643ba3d0fe0e1a03a44
I20260926 03:11:07.002200 20267 ts_tablet_manager.cc:1939] T c5ac7c3031e24643ba3d0fe0e1a03a44 P e6270617a04045a4b47ab30e53ec1410: Deleting consensus metadata
I20260926 03:11:07.003355 19752 catalog_manager.cc:4841] TS f6b3cefe617e4895b864ec10fe921174 (127.0.97.2:39169): tablet c5ac7c3031e24643ba3d0fe0e1a03a44 (table test-table [id=5274cab19af649f096bbc452b52ce6f4]) successfully deleted
I20260926 03:11:07.004096 19752 catalog_manager.cc:4841] TS e6270617a04045a4b47ab30e53ec1410 (127.0.97.3:38181): tablet c5ac7c3031e24643ba3d0fe0e1a03a44 (table test-table [id=5274cab19af649f096bbc452b52ce6f4]) successfully deleted
I20260926 03:11:07.007007 20240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:06.977499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54144 (local address 127.0.97.1:44237)
0926 03:11:06.977676 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:06.977682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:06.984140 (+ 6458us) server_negotiation.cc:407] Connection header received
0926 03:11:06.984316 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:06.984324 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:06.984430 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:06.984560 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:06.988704 (+ 4144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.989913 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.992427 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:06.993109 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:06.993244 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:06.998944 (+ 5700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:06.998965 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:06.998988 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:06.999047 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:07.003247 (+ 4200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:07.004514 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:07.004523 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:07.004525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:07.004677 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:07.005082 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:07.005086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:07.005088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:07.005616 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:07.006084 (+ 468us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:07.006326 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 03:11:07.006639 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:11:07.007959 19951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:11:07.114143 20268 ts_tablet_manager.cc:1918] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:11:07.114660 20268 log.cc:1196] T c5ac7c3031e24643ba3d0fe0e1a03a44 P fa0d108fdf5e474186ea89619406ea93: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391950058021-388-0/minicluster-data/ts-0/wal/wals/c5ac7c3031e24643ba3d0fe0e1a03a44
I20260926 03:11:07.114818 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 19828
W20260926 03:11:07.130167 19751 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44237 (error 108)
I20260926 03:11:07.130358 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 19962
I20260926 03:11:07.132143 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:07.131669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44237 (local address 127.0.0.1:54160)
0926 03:11:07.131839 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:07.131970 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 03:11:07.132372 19751 catalog_manager.cc:4584] TS fa0d108fdf5e474186ea89619406ea93 (127.0.97.1:44237): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c5ac7c3031e24643ba3d0fe0e1a03a44: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44237: connect: Connection refused (error 111)
Sep 26 03:11:07 dist-test-slave-hmbw krb5kdc[19722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:11:07.145816 20242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:07.131237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44762 (local address 127.0.97.2:39169)
0926 03:11:07.131388 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:07.131394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:07.131438 (+ 44us) server_negotiation.cc:407] Connection header received
0926 03:11:07.131656 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:07.131662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:07.131749 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:07.131863 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:07.133226 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:07.134007 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:07.135300 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:07.135721 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:07.135817 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:07.140051 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:07.140068 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:07.140086 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:07.140128 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:07.142998 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:07.143765 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:07.143773 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:07.143775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:07.143919 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:07.144274 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:07.144278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:07.144280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:07.144670 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:07.144966 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:07.145247 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 03:11:07.145479 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 03:11:07.146634 20085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:11:07.186363 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:07.185814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44237 (local address 127.0.0.1:54176)
0926 03:11:07.186081 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:07.186192 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260926 03:11:07.187639 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:07.187149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44237 (local address 127.0.0.1:54178)
0926 03:11:07.187348 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:07.187491 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 03:11:07.237227 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 19962
I20260926 03:11:07.248242 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:07.247676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44237 (local address 127.0.0.1:54194)
0926 03:11:07.247911 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:07.248027 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 03:11:07.249372 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:07.248965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44237 (local address 127.0.0.1:54210)
0926 03:11:07.249130 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:07.249232 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 03:11:07.251765 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 20096
Sep 26 03:11:07 dist-test-slave-hmbw krb5kdc[19722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:11:07.268599 20239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:07.252588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56852 (local address 127.0.97.3:38181)
0926 03:11:07.252817 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:07.252822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:07.252842 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:11:07.252898 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:07.252902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:07.252968 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:07.253068 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:07.254494 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:07.255332 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:07.256782 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:07.257207 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:07.257438 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:07.262275 (+ 4837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:07.262367 (+ 92us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:07.262384 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:07.262429 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:07.265526 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:07.266372 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:07.266382 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:07.266384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:07.266648 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:07.267073 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:07.267077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:07.267080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:07.267473 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:07.267707 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:07.268061 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 03:11:07.268249 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 03:11:07.269320 20219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:11:07.330086 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:07.329583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44237 (local address 127.0.0.1:54220)
0926 03:11:07.329841 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:07.329933 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 03:11:07.331559 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:07.331024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44237 (local address 127.0.0.1:54234)
0926 03:11:07.331334 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:07.331422 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260926 03:11:07.361258 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 20096
I20260926 03:11:07.375115 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 19735
I20260926 03:11:07.376255 19797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:11:07.490222 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 19735
2026-09-26T03:11:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4221 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T03:11:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:11:07Z Disabled control of system clock
I20260926 03:11:07.562773 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37027
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:11:07.913533 20279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:07.913854 20279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:07.913962 20279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:07.921569 20279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:11:07.921694 20279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:07.921732 20279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:07.921818 20279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:11:07.921880 20279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:11:07.931587 20279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37027
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:07.933007 20279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:07.935056 20279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:07.948472 20287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:07.948412 20285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:07.948561 20284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:07.949381 20279 server_base.cc:1034] running on GCE node
I20260926 03:11:07.950261 20279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:07.951874 20279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:07.953320 20279 hybrid_clock.cc:648] HybridClock initialized: now 1790392267953220 us; error 51 us; skew 500 ppm
I20260926 03:11:07.954033 20279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:11:07.956844 20279 webserver.cc:460] Webserver started at http://127.0.97.62:33471/ using document root <none> and password file <none>
I20260926 03:11:07.957599 20279 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:07.957697 20279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:07.957999 20279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:07.961158 20279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "5ec782d7ce78412eb045156d1066cf44"
format_stamp: "Formatted at 2026-09-26 03:11:07 on dist-test-slave-hmbw"
server_key: "0a887b634206790e5c50771303c96b79"
server_key_iv: "d817f14a2291aee5d4ea809d0b58a490"
server_key_version: "encryptionkey@0"
I20260926 03:11:07.961951 20279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "5ec782d7ce78412eb045156d1066cf44"
format_stamp: "Formatted at 2026-09-26 03:11:07 on dist-test-slave-hmbw"
server_key: "0a887b634206790e5c50771303c96b79"
server_key_iv: "d817f14a2291aee5d4ea809d0b58a490"
server_key_version: "encryptionkey@0"
I20260926 03:11:07.967737 20279 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 03:11:07.971827 20293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:07.973580 20279 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 03:11:07.973768 20279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "5ec782d7ce78412eb045156d1066cf44"
format_stamp: "Formatted at 2026-09-26 03:11:07 on dist-test-slave-hmbw"
server_key: "0a887b634206790e5c50771303c96b79"
server_key_iv: "d817f14a2291aee5d4ea809d0b58a490"
server_key_version: "encryptionkey@0"
I20260926 03:11:07.973995 20279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:08.004232 20279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:08.005106 20279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:08.005355 20279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:08.024717 20279 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37027
I20260926 03:11:08.024920 20344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37027 every 8 connection(s)
I20260926 03:11:08.026307 20279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:11:08.030655 20345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:08.031828 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 20279
I20260926 03:11:08.031998 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:11:08.032544 388 external_mini_cluster.cc:1428] Setting key 20a25149682c5324767a5d3929e34153
I20260926 03:11:08.046741 20345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec782d7ce78412eb045156d1066cf44: Bootstrap starting.
I20260926 03:11:08.049461 20348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:08.035184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52306 (local address 127.0.97.62:37027)
0926 03:11:08.036705 (+ 1521us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:08.036718 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:08.036769 (+ 51us) server_negotiation.cc:407] Connection header received
0926 03:11:08.038453 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:08.038482 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:08.039125 (+ 643us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:08.039517 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:08.040179 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:08.041558 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:08.043700 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:08.044902 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:08.045090 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:08.046588 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:08.046600 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:08.046617 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:08.047235 (+ 618us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:08.047326 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:08.047361 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:08.047463 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:08.047783 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 03:11:08.048334 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":839,"thread_start_us":528,"threads_started":1}
I20260926 03:11:08.050786 20345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ec782d7ce78412eb045156d1066cf44: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:08.051956 20345 log.cc:824] T 00000000000000000000000000000000 P 5ec782d7ce78412eb045156d1066cf44: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:08.055135 20345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec782d7ce78412eb045156d1066cf44: No bootstrap required, opened a new log
I20260926 03:11:08.062855 20345 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ec782d7ce78412eb045156d1066cf44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec782d7ce78412eb045156d1066cf44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37027 } }
I20260926 03:11:08.063262 20345 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ec782d7ce78412eb045156d1066cf44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:08.063359 20345 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ec782d7ce78412eb045156d1066cf44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec782d7ce78412eb045156d1066cf44, State: Initialized, Role: FOLLOWER
I20260926 03:11:08.063969 20345 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ec782d7ce78412eb045156d1066cf44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec782d7ce78412eb045156d1066cf44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37027 } }
I20260926 03:11:08.064198 20345 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ec782d7ce78412eb045156d1066cf44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:11:08.064323 20345 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ec782d7ce78412eb045156d1066cf44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:11:08.064481 20345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ec782d7ce78412eb045156d1066cf44 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:08.067626 20345 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ec782d7ce78412eb045156d1066cf44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec782d7ce78412eb045156d1066cf44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37027 } }
I20260926 03:11:08.068193 20345 leader_election.cc:302] T 00000000000000000000000000000000 P 5ec782d7ce78412eb045156d1066cf44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ec782d7ce78412eb045156d1066cf44; no voters:
I20260926 03:11:08.068861 20345 leader_election.cc:288] T 00000000000000000000000000000000 P 5ec782d7ce78412eb045156d1066cf44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:11:08.069025 20350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ec782d7ce78412eb045156d1066cf44 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:11:08.069831 20350 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ec782d7ce78412eb045156d1066cf44 [term 1 LEADER]: Becoming Leader. State: Replica: 5ec782d7ce78412eb045156d1066cf44, State: Running, Role: LEADER
I20260926 03:11:08.070420 20350 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ec782d7ce78412eb045156d1066cf44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec782d7ce78412eb045156d1066cf44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37027 } }
I20260926 03:11:08.071084 20345 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ec782d7ce78412eb045156d1066cf44 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:11:08.073381 20351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec782d7ce78412eb045156d1066cf44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ec782d7ce78412eb045156d1066cf44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec782d7ce78412eb045156d1066cf44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37027 } } }
I20260926 03:11:08.073822 20351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec782d7ce78412eb045156d1066cf44 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:08.074301 20352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec782d7ce78412eb045156d1066cf44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ec782d7ce78412eb045156d1066cf44. Latest consensus state: current_term: 1 leader_uuid: "5ec782d7ce78412eb045156d1066cf44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec782d7ce78412eb045156d1066cf44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37027 } } }
I20260926 03:11:08.074795 20352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec782d7ce78412eb045156d1066cf44 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:08.083185 20357 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:11:08.088121 20357 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:11:08.097925 20357 catalog_manager.cc:1347] Generated new cluster ID: 84676eff900f4461a7fa4d8b94308937
I20260926 03:11:08.098069 20357 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:11:08.123342 20357 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:11:08.124718 20357 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:11:08.138655 20357 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ec782d7ce78412eb045156d1066cf44: Generated new TSK 0
I20260926 03:11:08.139635 20357 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:11:08.149570 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37027
--builtin_ntp_servers=127.0.97.20:43771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:11:08.526822 20369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:08.527143 20369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:08.527258 20369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:08.534703 20369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:08.534835 20369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:08.534974 20369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:11:08.544500 20369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:08.545805 20369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:08.547711 20369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:08.561831 20374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:08.561805 20375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:08.561867 20377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:08.562680 20369 server_base.cc:1034] running on GCE node
I20260926 03:11:08.563350 20369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:08.564348 20369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:08.565588 20369 hybrid_clock.cc:648] HybridClock initialized: now 1790392268565518 us; error 57 us; skew 500 ppm
I20260926 03:11:08.565908 20369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:11:08.568452 20369 webserver.cc:460] Webserver started at http://127.0.97.1:38181/ using document root <none> and password file <none>
I20260926 03:11:08.569137 20369 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:08.569224 20369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:08.569507 20369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:08.572548 20369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-0/data/instance:
uuid: "5204eb8328ac433fa14e81e0a0d549cf"
format_stamp: "Formatted at 2026-09-26 03:11:08 on dist-test-slave-hmbw"
server_key: "06175f271fa4e84333004ec5489dc4dd"
server_key_iv: "7136ed6893969dded154a3b4db0b085c"
server_key_version: "encryptionkey@0"
I20260926 03:11:08.573308 20369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5204eb8328ac433fa14e81e0a0d549cf"
format_stamp: "Formatted at 2026-09-26 03:11:08 on dist-test-slave-hmbw"
server_key: "06175f271fa4e84333004ec5489dc4dd"
server_key_iv: "7136ed6893969dded154a3b4db0b085c"
server_key_version: "encryptionkey@0"
I20260926 03:11:08.579067 20369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:11:08.583093 20383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:08.584794 20369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:11:08.585026 20369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-0/wal
uuid: "5204eb8328ac433fa14e81e0a0d549cf"
format_stamp: "Formatted at 2026-09-26 03:11:08 on dist-test-slave-hmbw"
server_key: "06175f271fa4e84333004ec5489dc4dd"
server_key_iv: "7136ed6893969dded154a3b4db0b085c"
server_key_version: "encryptionkey@0"
I20260926 03:11:08.585345 20369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:08.608783 20369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:08.609632 20369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:08.609881 20369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:08.610917 20369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:11:08.612445 20369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:11:08.612531 20369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:08.612612 20369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:11:08.612663 20369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:08.644186 20369 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46247
I20260926 03:11:08.644249 20495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46247 every 8 connection(s)
I20260926 03:11:08.645623 20369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:11:08.649711 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 20369
I20260926 03:11:08.649921 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:11:08.650458 388 external_mini_cluster.cc:1428] Setting key 2c3d750d358ec269192a64ef62b7eef7
I20260926 03:11:08.653533 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37027
--builtin_ntp_servers=127.0.97.20:43771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:11:08.660055 20498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:08.648463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45019 (local address 127.0.97.62:37027)
0926 03:11:08.648940 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:08.648944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:08.650152 (+ 1208us) server_negotiation.cc:407] Connection header received
0926 03:11:08.652533 (+ 2381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:08.652538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:08.652619 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:08.652720 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:08.654011 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:08.655114 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:08.656890 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:08.657443 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:08.657564 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:08.659110 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:08.659116 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:08.659120 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:08.659147 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:08.659193 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:08.659208 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:08.659297 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:08.659592 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 03:11:08.659706 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":252,"threads_started":1}
I20260926 03:11:08.661084 20499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:08.648873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37027 (local address 127.0.97.1:45019)
0926 03:11:08.649967 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:08.650028 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:08.652222 (+ 2194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:08.652993 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:08.653003 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:08.653112 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:11:08.653819 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:08.653851 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:08.655300 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:08.655306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:08.656702 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:08.656719 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:08.657710 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:08.657716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:08.657965 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:08.658753 (+ 788us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:11:08.658755 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:11:08.658801 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:11:08.658903 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:11:08.659371 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:11:08.659381 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:11:08.659389 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 03:11:08.659565 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 03:11:08.659930 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":282,"threads_started":1}
I20260926 03:11:08.662951 20496 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37027
I20260926 03:11:08.663313 20496 heartbeater.cc:464] Registering TS with master...
I20260926 03:11:08.664278 20496 heartbeater.cc:511] Master 127.0.97.62:37027 requested a full tablet report, sending...
I20260926 03:11:08.666949 20310 ts_manager.cc:194] Registered new tserver with Master: 5204eb8328ac433fa14e81e0a0d549cf (127.0.97.1:46247)
I20260926 03:11:08.668501 20310 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:45019
W20260926 03:11:08.988869 20500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:08.989208 20500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:08.989331 20500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:08.996874 20500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:08.996996 20500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:08.997148 20500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:11:09.006829 20500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:09.008354 20500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:09.010349 20500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:09.024593 20505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:09.024641 20506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:09.025192 20500 server_base.cc:1034] running on GCE node
W20260926 03:11:09.025435 20508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:09.026175 20500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:09.027282 20500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:09.028550 20500 hybrid_clock.cc:648] HybridClock initialized: now 1790392269028493 us; error 47 us; skew 500 ppm
I20260926 03:11:09.028918 20500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:11:09.031462 20500 webserver.cc:460] Webserver started at http://127.0.97.2:38479/ using document root <none> and password file <none>
I20260926 03:11:09.032233 20500 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:09.032335 20500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:09.032649 20500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:09.036361 20500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-1/data/instance:
uuid: "1e5f7ee088994d97bcfd941781cd9e2f"
format_stamp: "Formatted at 2026-09-26 03:11:09 on dist-test-slave-hmbw"
server_key: "9a40d693a6a47c702778b326c6bd5f76"
server_key_iv: "f2fd182d6e465b05997860c050cf844e"
server_key_version: "encryptionkey@0"
I20260926 03:11:09.037592 20500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1e5f7ee088994d97bcfd941781cd9e2f"
format_stamp: "Formatted at 2026-09-26 03:11:09 on dist-test-slave-hmbw"
server_key: "9a40d693a6a47c702778b326c6bd5f76"
server_key_iv: "f2fd182d6e465b05997860c050cf844e"
server_key_version: "encryptionkey@0"
I20260926 03:11:09.043826 20500 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 03:11:09.047811 20514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:09.049455 20500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:11:09.049654 20500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-1/wal
uuid: "1e5f7ee088994d97bcfd941781cd9e2f"
format_stamp: "Formatted at 2026-09-26 03:11:09 on dist-test-slave-hmbw"
server_key: "9a40d693a6a47c702778b326c6bd5f76"
server_key_iv: "f2fd182d6e465b05997860c050cf844e"
server_key_version: "encryptionkey@0"
I20260926 03:11:09.049880 20500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:09.072458 20500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:09.073340 20500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:09.073616 20500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:09.074755 20500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:11:09.076429 20500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:11:09.076534 20500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:09.076644 20500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:11:09.076710 20500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:09.108443 20500 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38713
I20260926 03:11:09.108505 20626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38713 every 8 connection(s)
I20260926 03:11:09.109700 20500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:11:09.111522 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 20500
I20260926 03:11:09.111688 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:11:09.112123 388 external_mini_cluster.cc:1428] Setting key b06afcb98c8e565a0d52990cec97755c
I20260926 03:11:09.115434 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37027
--builtin_ntp_servers=127.0.97.20:43771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:11:09.124435 20498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:09.112507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44087 (local address 127.0.97.62:37027)
0926 03:11:09.112727 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:09.112733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:09.114465 (+ 1732us) server_negotiation.cc:407] Connection header received
0926 03:11:09.116361 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:09.116368 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:09.116459 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:09.116570 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:09.118065 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.119208 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.121231 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.121798 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.121940 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:09.123662 (+ 1722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:09.123668 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:09.123671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:09.123694 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:09.123730 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:09.123743 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:09.123824 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:09.124088 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 03:11:09.124180 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:11:09.125849 20629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:09.113014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37027 (local address 127.0.97.2:44087)
0926 03:11:09.114189 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:09.114278 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:09.116050 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:09.116952 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:09.116963 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:09.117071 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:11:09.117795 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:09.117816 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.119426 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.119431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:09.121042 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:09.121061 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.122062 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.122068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:09.122364 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:09.123189 (+ 825us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:11:09.123193 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:11:09.123268 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:11:09.123394 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:11:09.123846 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:11:09.123856 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:11:09.123864 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 03:11:09.124073 (+ 209us) client_negotiation.cc:240] Negotiation successful
0926 03:11:09.124522 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":306,"threads_started":1}
I20260926 03:11:09.127496 20627 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37027
I20260926 03:11:09.127815 20627 heartbeater.cc:464] Registering TS with master...
I20260926 03:11:09.128664 20627 heartbeater.cc:511] Master 127.0.97.62:37027 requested a full tablet report, sending...
I20260926 03:11:09.130210 20310 ts_manager.cc:194] Registered new tserver with Master: 1e5f7ee088994d97bcfd941781cd9e2f (127.0.97.2:38713)
I20260926 03:11:09.131062 20310 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:44087
W20260926 03:11:09.453289 20630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:09.453589 20630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:09.453677 20630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:09.461251 20630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:09.461381 20630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:09.461524 20630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:11:09.471243 20630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:09.472621 20630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:09.474428 20630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:09.488113 20635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:09.488072 20636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:09.488075 20638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:09.489549 20630 server_base.cc:1034] running on GCE node
I20260926 03:11:09.490337 20630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:09.491587 20630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:09.492841 20630 hybrid_clock.cc:648] HybridClock initialized: now 1790392269492758 us; error 76 us; skew 500 ppm
I20260926 03:11:09.493199 20630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:11:09.495757 20630 webserver.cc:460] Webserver started at http://127.0.97.3:45147/ using document root <none> and password file <none>
I20260926 03:11:09.496497 20630 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:09.496591 20630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:09.496912 20630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:09.500149 20630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-2/data/instance:
uuid: "33308950f5a049239f9d10915b3d1b5b"
format_stamp: "Formatted at 2026-09-26 03:11:09 on dist-test-slave-hmbw"
server_key: "337b141a38d9a1334685678298cfcbb3"
server_key_iv: "21f2022c06a9018bd3690b94f7f04286"
server_key_version: "encryptionkey@0"
I20260926 03:11:09.501030 20630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-2/wal/instance:
uuid: "33308950f5a049239f9d10915b3d1b5b"
format_stamp: "Formatted at 2026-09-26 03:11:09 on dist-test-slave-hmbw"
server_key: "337b141a38d9a1334685678298cfcbb3"
server_key_iv: "21f2022c06a9018bd3690b94f7f04286"
server_key_version: "encryptionkey@0"
I20260926 03:11:09.506688 20630 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:11:09.510718 20644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:09.512334 20630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:11:09.512496 20630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-2/wal
uuid: "33308950f5a049239f9d10915b3d1b5b"
format_stamp: "Formatted at 2026-09-26 03:11:09 on dist-test-slave-hmbw"
server_key: "337b141a38d9a1334685678298cfcbb3"
server_key_iv: "21f2022c06a9018bd3690b94f7f04286"
server_key_version: "encryptionkey@0"
I20260926 03:11:09.512705 20630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:09.533007 20630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:09.533893 20630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:09.534149 20630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:09.535250 20630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:11:09.536823 20630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:11:09.536914 20630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:09.536988 20630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:11:09.537043 20630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:09.570179 20630 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40199
I20260926 03:11:09.570200 20756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40199 every 8 connection(s)
I20260926 03:11:09.571702 20630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:11:09.573083 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 20630
I20260926 03:11:09.573269 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:11:09.573830 388 external_mini_cluster.cc:1428] Setting key 19513e3012f38b196caf4da8b2e5e199
I20260926 03:11:09.584868 20498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:09.574814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34637 (local address 127.0.97.62:37027)
0926 03:11:09.575054 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:09.575060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:09.576363 (+ 1303us) server_negotiation.cc:407] Connection header received
0926 03:11:09.578337 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:09.578342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:09.578432 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:09.578539 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:09.579656 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.580395 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.582157 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.582634 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.582762 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:09.584117 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:09.584122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:09.584126 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:09.584144 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:09.584178 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:09.584190 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:09.584264 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:09.584501 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 03:11:09.584594 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:11:09.585855 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:09.575259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37027 (local address 127.0.97.3:34637)
0926 03:11:09.576209 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:09.576259 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:09.578069 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:09.578776 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:09.578784 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:09.578878 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:11:09.579494 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:09.579512 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.580543 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.580547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:09.582025 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:09.582039 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.582866 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.582874 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:09.583119 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:09.583764 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:11:09.583766 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:11:09.583807 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:11:09.583905 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:11:09.584273 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:11:09.584279 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:11:09.584285 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 03:11:09.584487 (+ 202us) client_negotiation.cc:240] Negotiation successful
0926 03:11:09.584803 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":297,"threads_started":1}
I20260926 03:11:09.587160 20757 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37027
I20260926 03:11:09.587482 20757 heartbeater.cc:464] Registering TS with master...
I20260926 03:11:09.588380 20757 heartbeater.cc:511] Master 127.0.97.62:37027 requested a full tablet report, sending...
I20260926 03:11:09.589951 20310 ts_manager.cc:194] Registered new tserver with Master: 33308950f5a049239f9d10915b3d1b5b (127.0.97.3:40199)
I20260926 03:11:09.590801 20310 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:34637
I20260926 03:11:09.602216 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:11:09.614130 20498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:09.607294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52310 (local address 127.0.97.62:37027)
0926 03:11:09.607618 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:09.607625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:09.608123 (+ 498us) server_negotiation.cc:407] Connection header received
0926 03:11:09.608320 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:09.608327 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:09.608423 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:09.608516 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:09.609043 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.609745 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.611202 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.611704 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.611824 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:09.613349 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:09.613354 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:09.613357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:09.613378 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:09.613414 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:09.613426 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:09.613588 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:09.613708 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 03:11:09.613818 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:11:09.619531 20310 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:11:09.628003 20310 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:11:09.652335 20769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:09.641058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46247 (local address 127.0.0.1:38932)
0926 03:11:09.641894 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:09.641942 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:09.642141 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:09.643516 (+ 1375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:09.643526 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:09.643562 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:11:09.644009 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:09.644022 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.646649 (+ 2627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.646656 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:09.648962 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:09.648983 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.651313 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.651321 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:09.651581 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:09.651645 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 03:11:09.651780 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 03:11:09.652019 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"threads_started":1}
I20260926 03:11:09.654134 20771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:09.641266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38932 (local address 127.0.97.1:46247)
0926 03:11:09.642831 (+ 1565us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:09.642840 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:09.642869 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:11:09.642976 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:09.642985 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:09.643259 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:09.644300 (+ 1041us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:11:09.644575 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.646426 (+ 1851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.649287 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.651037 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.653042 (+ 2005us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:09.653372 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:09.653518 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 03:11:09.653715 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1346,"thread_start_us":1210,"threads_started":1}
I20260926 03:11:09.658437 20431 tablet_service.cc:1467] Processing CreateTablet for tablet 967c6d17509d47d1905987ce05573344 (DEFAULT_TABLE table=test-table [id=94554fa8a3574692a2a7ded26cc26451]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:11:09.660624 20431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 967c6d17509d47d1905987ce05573344. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:09.660686 20769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:09.641151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40199 (local address 127.0.0.1:56482)
0926 03:11:09.652591 (+ 11440us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:09.652611 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:09.652722 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:09.654010 (+ 1288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:09.654015 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:09.654038 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:11:09.654212 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:09.654222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.656253 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.656258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:09.657619 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:09.657638 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.659886 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.659892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:09.660087 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:09.660120 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:11:09.660204 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 03:11:09.660347 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11316,"thread_start_us":687,"threads_started":1}
I20260926 03:11:09.660737 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:09.641668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56482 (local address 127.0.97.3:40199)
0926 03:11:09.652668 (+ 11000us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:09.652677 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:09.652718 (+ 41us) server_negotiation.cc:407] Connection header received
0926 03:11:09.653450 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:09.653460 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:09.653758 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:09.654644 (+ 886us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:11:09.654915 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.656110 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.657846 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.659663 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.659844 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:09.660178 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:09.660255 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 03:11:09.660418 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10824,"thread_start_us":10660,"threads_started":1}
I20260926 03:11:09.662245 20770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:09.653305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38713 (local address 127.0.0.1:41180)
0926 03:11:09.653806 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:09.653832 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:09.653945 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:09.654975 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:09.654979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:09.655004 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:11:09.655326 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:09.655339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.657171 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.657176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:09.659558 (+ 2382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:09.659579 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.661511 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.661518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:09.661691 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:09.661725 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 03:11:09.661821 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 03:11:09.661968 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352}
I20260926 03:11:09.662410 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:09.653502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41180 (local address 127.0.97.2:38713)
0926 03:11:09.654486 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:09.654513 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:09.654542 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:11:09.654625 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:09.654630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:09.654796 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:09.654953 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:11:09.655538 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.657012 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.659816 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.661290 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.661444 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:09.661693 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:09.661885 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 03:11:09.662074 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":620,"threads_started":1}
I20260926 03:11:09.664605 20692 tablet_service.cc:1467] Processing CreateTablet for tablet 967c6d17509d47d1905987ce05573344 (DEFAULT_TABLE table=test-table [id=94554fa8a3574692a2a7ded26cc26451]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:11:09.666200 20692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 967c6d17509d47d1905987ce05573344. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:09.666268 20562 tablet_service.cc:1467] Processing CreateTablet for tablet 967c6d17509d47d1905987ce05573344 (DEFAULT_TABLE table=test-table [id=94554fa8a3574692a2a7ded26cc26451]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:11:09.667766 20562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 967c6d17509d47d1905987ce05573344. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:09.672724 20496 heartbeater.cc:503] Master 127.0.97.62:37027 was elected leader, sending a full tablet report...
I20260926 03:11:09.675740 20774 tablet_bootstrap.cc:492] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf: Bootstrap starting.
I20260926 03:11:09.679190 20774 tablet_bootstrap.cc:654] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:09.680256 20774 log.cc:824] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:09.683476 20774 tablet_bootstrap.cc:492] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf: No bootstrap required, opened a new log
I20260926 03:11:09.683486 20777 tablet_bootstrap.cc:492] T 967c6d17509d47d1905987ce05573344 P 1e5f7ee088994d97bcfd941781cd9e2f: Bootstrap starting.
I20260926 03:11:09.683761 20774 ts_tablet_manager.cc:1397] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260926 03:11:09.684639 20776 tablet_bootstrap.cc:492] T 967c6d17509d47d1905987ce05573344 P 33308950f5a049239f9d10915b3d1b5b: Bootstrap starting.
I20260926 03:11:09.687258 20777 tablet_bootstrap.cc:654] T 967c6d17509d47d1905987ce05573344 P 1e5f7ee088994d97bcfd941781cd9e2f: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:09.688112 20776 tablet_bootstrap.cc:654] T 967c6d17509d47d1905987ce05573344 P 33308950f5a049239f9d10915b3d1b5b: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:09.688544 20777 log.cc:824] T 967c6d17509d47d1905987ce05573344 P 1e5f7ee088994d97bcfd941781cd9e2f: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:09.689415 20776 log.cc:824] T 967c6d17509d47d1905987ce05573344 P 33308950f5a049239f9d10915b3d1b5b: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:09.691419 20774 raft_consensus.cc:348] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5204eb8328ac433fa14e81e0a0d549cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46247 } } peers { permanent_uuid: "33308950f5a049239f9d10915b3d1b5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40199 } } peers { permanent_uuid: "1e5f7ee088994d97bcfd941781cd9e2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38713 } }
I20260926 03:11:09.691778 20774 raft_consensus.cc:374] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:09.691895 20774 raft_consensus.cc:729] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5204eb8328ac433fa14e81e0a0d549cf, State: Initialized, Role: FOLLOWER
I20260926 03:11:09.692471 20774 consensus_queue.cc:260] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5204eb8328ac433fa14e81e0a0d549cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46247 } } peers { permanent_uuid: "33308950f5a049239f9d10915b3d1b5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40199 } } peers { permanent_uuid: "1e5f7ee088994d97bcfd941781cd9e2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38713 } }
I20260926 03:11:09.694208 20774 ts_tablet_manager.cc:1428] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf: Time spent starting tablet: real 0.010s user 0.007s sys 0.002s
I20260926 03:11:09.697125 20777 tablet_bootstrap.cc:492] T 967c6d17509d47d1905987ce05573344 P 1e5f7ee088994d97bcfd941781cd9e2f: No bootstrap required, opened a new log
I20260926 03:11:09.697412 20777 ts_tablet_manager.cc:1397] T 967c6d17509d47d1905987ce05573344 P 1e5f7ee088994d97bcfd941781cd9e2f: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.012s
I20260926 03:11:09.697448 20776 tablet_bootstrap.cc:492] T 967c6d17509d47d1905987ce05573344 P 33308950f5a049239f9d10915b3d1b5b: No bootstrap required, opened a new log
I20260926 03:11:09.697731 20776 ts_tablet_manager.cc:1397] T 967c6d17509d47d1905987ce05573344 P 33308950f5a049239f9d10915b3d1b5b: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260926 03:11:09.705572 20776 raft_consensus.cc:348] T 967c6d17509d47d1905987ce05573344 P 33308950f5a049239f9d10915b3d1b5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5204eb8328ac433fa14e81e0a0d549cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46247 } } peers { permanent_uuid: "33308950f5a049239f9d10915b3d1b5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40199 } } peers { permanent_uuid: "1e5f7ee088994d97bcfd941781cd9e2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38713 } }
I20260926 03:11:09.705572 20777 raft_consensus.cc:348] T 967c6d17509d47d1905987ce05573344 P 1e5f7ee088994d97bcfd941781cd9e2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5204eb8328ac433fa14e81e0a0d549cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46247 } } peers { permanent_uuid: "33308950f5a049239f9d10915b3d1b5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40199 } } peers { permanent_uuid: "1e5f7ee088994d97bcfd941781cd9e2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38713 } }
I20260926 03:11:09.705960 20776 raft_consensus.cc:374] T 967c6d17509d47d1905987ce05573344 P 33308950f5a049239f9d10915b3d1b5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:09.705968 20777 raft_consensus.cc:374] T 967c6d17509d47d1905987ce05573344 P 1e5f7ee088994d97bcfd941781cd9e2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:09.706048 20776 raft_consensus.cc:729] T 967c6d17509d47d1905987ce05573344 P 33308950f5a049239f9d10915b3d1b5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33308950f5a049239f9d10915b3d1b5b, State: Initialized, Role: FOLLOWER
I20260926 03:11:09.706053 20777 raft_consensus.cc:729] T 967c6d17509d47d1905987ce05573344 P 1e5f7ee088994d97bcfd941781cd9e2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5f7ee088994d97bcfd941781cd9e2f, State: Initialized, Role: FOLLOWER
I20260926 03:11:09.706670 20777 consensus_queue.cc:260] T 967c6d17509d47d1905987ce05573344 P 1e5f7ee088994d97bcfd941781cd9e2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5204eb8328ac433fa14e81e0a0d549cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46247 } } peers { permanent_uuid: "33308950f5a049239f9d10915b3d1b5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40199 } } peers { permanent_uuid: "1e5f7ee088994d97bcfd941781cd9e2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38713 } }
I20260926 03:11:09.706670 20776 consensus_queue.cc:260] T 967c6d17509d47d1905987ce05573344 P 33308950f5a049239f9d10915b3d1b5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5204eb8328ac433fa14e81e0a0d549cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46247 } } peers { permanent_uuid: "33308950f5a049239f9d10915b3d1b5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40199 } } peers { permanent_uuid: "1e5f7ee088994d97bcfd941781cd9e2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38713 } }
I20260926 03:11:09.707881 20757 heartbeater.cc:503] Master 127.0.97.62:37027 was elected leader, sending a full tablet report...
I20260926 03:11:09.708324 20627 heartbeater.cc:503] Master 127.0.97.62:37027 was elected leader, sending a full tablet report...
I20260926 03:11:09.708659 20776 ts_tablet_manager.cc:1428] T 967c6d17509d47d1905987ce05573344 P 33308950f5a049239f9d10915b3d1b5b: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260926 03:11:09.708884 20777 ts_tablet_manager.cc:1428] T 967c6d17509d47d1905987ce05573344 P 1e5f7ee088994d97bcfd941781cd9e2f: Time spent starting tablet: real 0.011s user 0.008s sys 0.005s
I20260926 03:11:09.730834 20780 raft_consensus.cc:482] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:11:09.731148 20780 raft_consensus.cc:504] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5204eb8328ac433fa14e81e0a0d549cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46247 } } peers { permanent_uuid: "33308950f5a049239f9d10915b3d1b5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40199 } } peers { permanent_uuid: "1e5f7ee088994d97bcfd941781cd9e2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38713 } }
I20260926 03:11:09.732810 20780 leader_election.cc:288] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33308950f5a049239f9d10915b3d1b5b (127.0.97.3:40199), 1e5f7ee088994d97bcfd941781cd9e2f (127.0.97.2:38713)
I20260926 03:11:09.741866 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:09.733333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56621 (local address 127.0.97.3:40199)
0926 03:11:09.733540 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:09.733546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:09.734096 (+ 550us) server_negotiation.cc:407] Connection header received
0926 03:11:09.734237 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:09.734243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:09.734397 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:09.734489 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:11:09.735424 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.736497 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.739590 (+ 3093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.740834 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.740949 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:09.741050 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:09.741487 (+ 437us) server_negotiation.cc:299] Negotiation successful
0926 03:11:09.741610 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:11:09.741904 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:09.733333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41045 (local address 127.0.97.2:38713)
0926 03:11:09.733545 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:09.733552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:09.734009 (+ 457us) server_negotiation.cc:407] Connection header received
0926 03:11:09.734240 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:09.734246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:09.734343 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:09.734461 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:11:09.735391 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.736498 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.739599 (+ 3101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.740795 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.740949 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:09.741047 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:09.741465 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 03:11:09.741594 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 03:11:09.742019 20783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:09.733096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38713 (local address 127.0.97.1:41045)
0926 03:11:09.733878 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:09.733897 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:09.734065 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:09.734526 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:09.734532 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:09.734565 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:11:09.735216 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:09.735225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.736648 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.736654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:09.739343 (+ 2689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:09.739362 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.741042 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.741050 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:09.741267 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:09.741331 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 03:11:09.741435 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 03:11:09.741603 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":275,"thread_start_us":452,"threads_started":1}
I20260926 03:11:09.742038 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:09.733181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40199 (local address 127.0.97.1:56621)
0926 03:11:09.734009 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:09.734025 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:09.734119 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:09.734539 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:09.734543 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:09.734620 (+ 77us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:11:09.735186 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:09.735205 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.736650 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.736654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:09.739343 (+ 2689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:09.739362 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.741054 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.741060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:09.741266 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:09.741389 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 03:11:09.741461 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 03:11:09.741603 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":324,"spinlock_wait_cycles":51200,"thread_start_us":517,"threads_started":1}
I20260926 03:11:09.743276 20712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "967c6d17509d47d1905987ce05573344" candidate_uuid: "5204eb8328ac433fa14e81e0a0d549cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33308950f5a049239f9d10915b3d1b5b" is_pre_election: true
I20260926 03:11:09.743276 20582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "967c6d17509d47d1905987ce05573344" candidate_uuid: "5204eb8328ac433fa14e81e0a0d549cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5f7ee088994d97bcfd941781cd9e2f" is_pre_election: true
I20260926 03:11:09.743887 20712 raft_consensus.cc:2454] T 967c6d17509d47d1905987ce05573344 P 33308950f5a049239f9d10915b3d1b5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5204eb8328ac433fa14e81e0a0d549cf in term 0.
I20260926 03:11:09.743875 20582 raft_consensus.cc:2454] T 967c6d17509d47d1905987ce05573344 P 1e5f7ee088994d97bcfd941781cd9e2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5204eb8328ac433fa14e81e0a0d549cf in term 0.
I20260926 03:11:09.744706 20384 leader_election.cc:302] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5f7ee088994d97bcfd941781cd9e2f, 5204eb8328ac433fa14e81e0a0d549cf; no voters:
I20260926 03:11:09.745122 20780 raft_consensus.cc:2781] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:11:09.745280 20780 raft_consensus.cc:482] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:11:09.745373 20780 raft_consensus.cc:3037] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:09.747689 20780 raft_consensus.cc:504] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5204eb8328ac433fa14e81e0a0d549cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46247 } } peers { permanent_uuid: "33308950f5a049239f9d10915b3d1b5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40199 } } peers { permanent_uuid: "1e5f7ee088994d97bcfd941781cd9e2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38713 } }
I20260926 03:11:09.748414 20780 leader_election.cc:288] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf [CANDIDATE]: Term 1 election: Requested vote from peers 33308950f5a049239f9d10915b3d1b5b (127.0.97.3:40199), 1e5f7ee088994d97bcfd941781cd9e2f (127.0.97.2:38713)
I20260926 03:11:09.748895 20712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "967c6d17509d47d1905987ce05573344" candidate_uuid: "5204eb8328ac433fa14e81e0a0d549cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33308950f5a049239f9d10915b3d1b5b"
I20260926 03:11:09.748896 20582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "967c6d17509d47d1905987ce05573344" candidate_uuid: "5204eb8328ac433fa14e81e0a0d549cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5f7ee088994d97bcfd941781cd9e2f"
I20260926 03:11:09.749127 20712 raft_consensus.cc:3037] T 967c6d17509d47d1905987ce05573344 P 33308950f5a049239f9d10915b3d1b5b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:09.749123 20582 raft_consensus.cc:3037] T 967c6d17509d47d1905987ce05573344 P 1e5f7ee088994d97bcfd941781cd9e2f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:09.752463 20582 raft_consensus.cc:2454] T 967c6d17509d47d1905987ce05573344 P 1e5f7ee088994d97bcfd941781cd9e2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5204eb8328ac433fa14e81e0a0d549cf in term 1.
I20260926 03:11:09.752463 20712 raft_consensus.cc:2454] T 967c6d17509d47d1905987ce05573344 P 33308950f5a049239f9d10915b3d1b5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5204eb8328ac433fa14e81e0a0d549cf in term 1.
I20260926 03:11:09.753079 20386 leader_election.cc:302] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33308950f5a049239f9d10915b3d1b5b, 5204eb8328ac433fa14e81e0a0d549cf; no voters:
I20260926 03:11:09.753495 20780 raft_consensus.cc:2781] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:11:09.753979 20780 raft_consensus.cc:686] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf [term 1 LEADER]: Becoming Leader. State: Replica: 5204eb8328ac433fa14e81e0a0d549cf, State: Running, Role: LEADER
I20260926 03:11:09.754559 20780 consensus_queue.cc:237] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5204eb8328ac433fa14e81e0a0d549cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46247 } } peers { permanent_uuid: "33308950f5a049239f9d10915b3d1b5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40199 } } peers { permanent_uuid: "1e5f7ee088994d97bcfd941781cd9e2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38713 } }
I20260926 03:11:09.759196 20310 catalog_manager.cc:5495] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 5204eb8328ac433fa14e81e0a0d549cf (127.0.97.1). New cstate: current_term: 1 leader_uuid: "5204eb8328ac433fa14e81e0a0d549cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5204eb8328ac433fa14e81e0a0d549cf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33308950f5a049239f9d10915b3d1b5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5f7ee088994d97bcfd941781cd9e2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38713 } health_report { overall_health: UNKNOWN } } }
I20260926 03:11:09.785881 20771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:09.780462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38936 (local address 127.0.97.1:46247)
0926 03:11:09.780653 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:09.780659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:09.780676 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:11:09.780761 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:09.780764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:09.780861 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:09.780982 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:11:09.781297 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.782186 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.783773 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.784243 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.784364 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:09.784856 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:11:09.785371 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:11:09.785484 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:09.785586 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 03:11:09.785660 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 03:11:09.794529 20712 raft_consensus.cc:1261] T 967c6d17509d47d1905987ce05573344 P 33308950f5a049239f9d10915b3d1b5b [term 1 FOLLOWER]: Refusing update from remote peer 5204eb8328ac433fa14e81e0a0d549cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:11:09.794488 20582 raft_consensus.cc:1261] T 967c6d17509d47d1905987ce05573344 P 1e5f7ee088994d97bcfd941781cd9e2f [term 1 FOLLOWER]: Refusing update from remote peer 5204eb8328ac433fa14e81e0a0d549cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:11:09.795851 20785 consensus_queue.cc:1035] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e5f7ee088994d97bcfd941781cd9e2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:11:09.796279 20780 consensus_queue.cc:1035] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "33308950f5a049239f9d10915b3d1b5b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:11:09.815943 20788 mvcc.cc:205] Tried to move back new op lower bound from 7333446737067614208 to 7333446736918315008. Current Snapshot: MvccSnapshot[applied={T|T < 7333446737067614208}]
I20260926 03:11:09.833266 20310 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:11:09.833572 20310 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:11:09.837621 20310 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ec782d7ce78412eb045156d1066cf44: Sending DeleteTablet for 3 replicas of tablet 967c6d17509d47d1905987ce05573344
I20260926 03:11:09.839017 20692 tablet_service.cc:1514] Processing DeleteTablet for tablet 967c6d17509d47d1905987ce05573344 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:11:09 UTC) from {username='slave'} at 127.0.0.1:56482
I20260926 03:11:09.839017 20431 tablet_service.cc:1514] Processing DeleteTablet for tablet 967c6d17509d47d1905987ce05573344 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:11:09 UTC) from {username='slave'} at 127.0.0.1:38932
I20260926 03:11:09.839080 20562 tablet_service.cc:1514] Processing DeleteTablet for tablet 967c6d17509d47d1905987ce05573344 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:11:09 UTC) from {username='slave'} at 127.0.0.1:41180
I20260926 03:11:09.840391 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 20369
I20260926 03:11:09.858825 20799 tablet_replica.cc:330] T 967c6d17509d47d1905987ce05573344 P 33308950f5a049239f9d10915b3d1b5b: stopping tablet replica
I20260926 03:11:09.859534 20801 tablet_replica.cc:330] T 967c6d17509d47d1905987ce05573344 P 1e5f7ee088994d97bcfd941781cd9e2f: stopping tablet replica
I20260926 03:11:09.860270 20800 tablet_replica.cc:330] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf: stopping tablet replica
I20260926 03:11:09.860790 20800 raft_consensus.cc:2229] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:11:09.861202 20801 raft_consensus.cc:2229] T 967c6d17509d47d1905987ce05573344 P 1e5f7ee088994d97bcfd941781cd9e2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:11:09.861541 20771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:09.842076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38950 (local address 127.0.97.1:46247)
0926 03:11:09.842269 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:09.842277 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:09.847738 (+ 5461us) server_negotiation.cc:407] Connection header received
0926 03:11:09.848744 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:09.848750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:09.848900 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:09.849000 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:09.850325 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.851664 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.855750 (+ 4086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.856534 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.856657 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:09.859887 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:09.859901 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:09.859925 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:09.860630 (+ 705us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:09.860681 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:09.860703 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:09.860811 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:09.861022 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 03:11:09.861148 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:11:09.861649 20800 raft_consensus.cc:2258] T 967c6d17509d47d1905987ce05573344 P 5204eb8328ac433fa14e81e0a0d549cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:11:09.862234 20799 raft_consensus.cc:2229] T 967c6d17509d47d1905987ce05573344 P 33308950f5a049239f9d10915b3d1b5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:11:09.862640 20801 raft_consensus.cc:2258] T 967c6d17509d47d1905987ce05573344 P 1e5f7ee088994d97bcfd941781cd9e2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:11:09.862962 20491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:11:09.862932 20799 raft_consensus.cc:2258] T 967c6d17509d47d1905987ce05573344 P 33308950f5a049239f9d10915b3d1b5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:11:09.865504 20801 ts_tablet_manager.cc:1905] T 967c6d17509d47d1905987ce05573344 P 1e5f7ee088994d97bcfd941781cd9e2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:11:09.865875 20799 ts_tablet_manager.cc:1905] T 967c6d17509d47d1905987ce05573344 P 33308950f5a049239f9d10915b3d1b5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:11:09.871814 20801 ts_tablet_manager.cc:1918] T 967c6d17509d47d1905987ce05573344 P 1e5f7ee088994d97bcfd941781cd9e2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:11:09.871987 20801 log.cc:1196] T 967c6d17509d47d1905987ce05573344 P 1e5f7ee088994d97bcfd941781cd9e2f: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-1/wal/wals/967c6d17509d47d1905987ce05573344
I20260926 03:11:09.872493 20801 ts_tablet_manager.cc:1939] T 967c6d17509d47d1905987ce05573344 P 1e5f7ee088994d97bcfd941781cd9e2f: Deleting consensus metadata
I20260926 03:11:09.873119 20799 ts_tablet_manager.cc:1918] T 967c6d17509d47d1905987ce05573344 P 33308950f5a049239f9d10915b3d1b5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:11:09.873298 20799 log.cc:1196] T 967c6d17509d47d1905987ce05573344 P 33308950f5a049239f9d10915b3d1b5b: Deleting WAL directory at /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391950058021-388-0/minicluster-data/ts-2/wal/wals/967c6d17509d47d1905987ce05573344
I20260926 03:11:09.873819 20799 ts_tablet_manager.cc:1939] T 967c6d17509d47d1905987ce05573344 P 33308950f5a049239f9d10915b3d1b5b: Deleting consensus metadata
I20260926 03:11:09.874200 20294 catalog_manager.cc:4841] TS 1e5f7ee088994d97bcfd941781cd9e2f (127.0.97.2:38713): tablet 967c6d17509d47d1905987ce05573344 (table test-table [id=94554fa8a3574692a2a7ded26cc26451]) successfully deleted
I20260926 03:11:09.875617 20296 catalog_manager.cc:4841] TS 33308950f5a049239f9d10915b3d1b5b (127.0.97.3:40199): tablet 967c6d17509d47d1905987ce05573344 (table test-table [id=94554fa8a3574692a2a7ded26cc26451]) successfully deleted
I20260926 03:11:09.963033 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 20369
I20260926 03:11:09.978317 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 20500
W20260926 03:11:09.978456 20294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:46247 (error 108)
I20260926 03:11:09.980190 20770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:09.979788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46247 (local address 127.0.0.1:38960)
0926 03:11:09.979946 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:09.980054 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 03:11:09.980455 20294 catalog_manager.cc:4584] TS 5204eb8328ac433fa14e81e0a0d549cf (127.0.97.1:46247): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 967c6d17509d47d1905987ce05573344: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46247: connect: Connection refused (error 111)
I20260926 03:11:09.990146 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:09.979137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41182 (local address 127.0.97.2:38713)
0926 03:11:09.979317 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:09.979323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:09.979392 (+ 69us) server_negotiation.cc:407] Connection header received
0926 03:11:09.979628 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:09.979633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:09.979724 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:09.979832 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:09.980275 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.981281 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.982713 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:09.983115 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:09.983230 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:09.988697 (+ 5467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:09.988707 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:09.988724 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:09.989345 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:09.989402 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:09.989427 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:09.989537 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:09.989689 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 03:11:09.989846 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:11:09.990856 20622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:11:10.034345 20770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:10.033707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46247 (local address 127.0.0.1:38976)
0926 03:11:10.033974 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:10.034089 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260926 03:11:10.035533 20770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:10.035125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46247 (local address 127.0.0.1:38992)
0926 03:11:10.035293 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:10.035384 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 03:11:10.083585 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 20500
I20260926 03:11:10.096511 20770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:10.095906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46247 (local address 127.0.0.1:39000)
0926 03:11:10.096152 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:10.096257 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260926 03:11:10.097465 20770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:10.097120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46247 (local address 127.0.0.1:39014)
0926 03:11:10.097275 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:10.097353 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 03:11:10.097899 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 20630
I20260926 03:11:10.104825 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:10.098576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56498 (local address 127.0.97.3:40199)
0926 03:11:10.098758 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:10.098764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:10.098784 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:11:10.098977 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:10.098984 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:10.099084 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:10.099168 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:10.099635 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:10.100419 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:10.101785 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:10.102241 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:10.102413 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:10.103509 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:10.103518 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:10.103533 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:10.104103 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:10.104152 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:10.104179 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:10.104287 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:10.104410 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 03:11:10.104529 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 03:11:10.105535 20752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:11:10.178287 20770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:10.177744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46247 (local address 127.0.0.1:39026)
0926 03:11:10.177965 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:10.178078 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 03:11:10.179628 20770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:10.179144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46247 (local address 127.0.0.1:39030)
0926 03:11:10.179377 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:10.179467 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260926 03:11:10.194139 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 20630
I20260926 03:11:10.208210 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 20279
I20260926 03:11:10.209240 20340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:11:10.317682 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 20279
2026-09-26T03:11:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2826 ms)
[----------] 6 tests from EncryptionPolicyTest (21221 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:11:10 dist-test-slave-hmbw krb5kdc[20810](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:11:10 dist-test-slave-hmbw krb5kdc[20810](info): set up 2 sockets
Sep 26 03:11:10 dist-test-slave-hmbw krb5kdc[20810](info): commencing operation
krb5kdc: starting...
W20260926 03:11:11.354820 388 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.007s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:11:11 dist-test-slave-hmbw krb5kdc[20810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:11:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:11:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:11:11.609761 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37237
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:11:11.942986 20823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:11.943290 20823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:11.943418 20823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:11.951166 20823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:11:11.951315 20823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:11.951380 20823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:11.951467 20823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:11:11.951524 20823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:11:11.962474 20823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37237
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:11.963954 20823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:11.965281 20823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:11.977491 20831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:11.977496 20829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:11.978044 20828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:11.979063 20823 server_base.cc:1034] running on GCE node
I20260926 03:11:11.979998 20823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:11.981614 20823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:11.982862 20823 hybrid_clock.cc:648] HybridClock initialized: now 1790392271982808 us; error 53 us; skew 500 ppm
I20260926 03:11:11.983325 20823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:11:11 dist-test-slave-hmbw krb5kdc[20810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:11:11.987568 20823 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:11:11.989187 20823 webserver.cc:460] Webserver started at http://127.0.0.1:41871/ using document root <none> and password file <none>
I20260926 03:11:11.989902 20823 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:11.990010 20823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:11.990278 20823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:11.993428 20823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "f9b5a44a22ae454bbc45bd1f1db880b3"
format_stamp: "Formatted at 2026-09-26 03:11:11 on dist-test-slave-hmbw"
server_key: "3fa69ed52194f35a14aa71c769978d52"
server_key_iv: "bb194ccb3c904fcabcd51013e537b790"
server_key_version: "encryptionkey@0"
I20260926 03:11:11.994176 20823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "f9b5a44a22ae454bbc45bd1f1db880b3"
format_stamp: "Formatted at 2026-09-26 03:11:11 on dist-test-slave-hmbw"
server_key: "3fa69ed52194f35a14aa71c769978d52"
server_key_iv: "bb194ccb3c904fcabcd51013e537b790"
server_key_version: "encryptionkey@0"
I20260926 03:11:12.000766 20823 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260926 03:11:12.005672 20838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:12.007441 20823 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:11:12.007632 20823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "f9b5a44a22ae454bbc45bd1f1db880b3"
format_stamp: "Formatted at 2026-09-26 03:11:11 on dist-test-slave-hmbw"
server_key: "3fa69ed52194f35a14aa71c769978d52"
server_key_iv: "bb194ccb3c904fcabcd51013e537b790"
server_key_version: "encryptionkey@0"
I20260926 03:11:12.007834 20823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:12.020761 20823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:12.025223 20823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:12.025485 20823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:12.044301 20823 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37237
I20260926 03:11:12.044332 20889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37237 every 8 connection(s)
I20260926 03:11:12.046099 20823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:11:12.050957 20890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:12.054473 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 20823
I20260926 03:11:12.054711 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:11:12.055358 388 external_mini_cluster.cc:1428] Setting key 158cb4ff0bbed9703e805bed43bda778
I20260926 03:11:12.064568 20890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b5a44a22ae454bbc45bd1f1db880b3: Bootstrap starting.
I20260926 03:11:12.068562 20890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9b5a44a22ae454bbc45bd1f1db880b3: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:12.069964 20890 log.cc:824] T 00000000000000000000000000000000 P f9b5a44a22ae454bbc45bd1f1db880b3: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:11:12 dist-test-slave-hmbw krb5kdc[20810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 03:11:12.073899 20890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b5a44a22ae454bbc45bd1f1db880b3: No bootstrap required, opened a new log
I20260926 03:11:12.082350 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:12.057657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47045 (local address 127.0.0.1:37237)
0926 03:11:12.058476 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:12.058489 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:12.058576 (+ 87us) server_negotiation.cc:407] Connection header received
0926 03:11:12.060274 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:12.060301 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:12.060766 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:12.061143 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:12.063017 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:12.064420 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:12.066598 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:12.067333 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:12.067491 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:12.072888 (+ 5397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:12.072900 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:12.072924 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:12.072980 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:12.077096 (+ 4116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:12.078076 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:12.078087 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:12.078102 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:12.078323 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:12.078843 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:12.078849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:12.078853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:12.079625 (+ 772us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:12.079985 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:12.080429 (+ 444us) server_negotiation.cc:299] Negotiation successful
0926 03:11:12.080843 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":376,"threads_started":1}
I20260926 03:11:12.084800 20890 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9b5a44a22ae454bbc45bd1f1db880b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b5a44a22ae454bbc45bd1f1db880b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37237 } }
I20260926 03:11:12.085249 20890 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9b5a44a22ae454bbc45bd1f1db880b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:12.085482 20890 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9b5a44a22ae454bbc45bd1f1db880b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b5a44a22ae454bbc45bd1f1db880b3, State: Initialized, Role: FOLLOWER
I20260926 03:11:12.086138 20890 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9b5a44a22ae454bbc45bd1f1db880b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b5a44a22ae454bbc45bd1f1db880b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37237 } }
I20260926 03:11:12.086438 20890 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9b5a44a22ae454bbc45bd1f1db880b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:11:12.086593 20890 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9b5a44a22ae454bbc45bd1f1db880b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:11:12.086838 20890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9b5a44a22ae454bbc45bd1f1db880b3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:12.087173 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:11:12.089135 20890 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9b5a44a22ae454bbc45bd1f1db880b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b5a44a22ae454bbc45bd1f1db880b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37237 } }
I20260926 03:11:12.089669 20890 leader_election.cc:302] T 00000000000000000000000000000000 P f9b5a44a22ae454bbc45bd1f1db880b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9b5a44a22ae454bbc45bd1f1db880b3; no voters:
I20260926 03:11:12.090338 20890 leader_election.cc:288] T 00000000000000000000000000000000 P f9b5a44a22ae454bbc45bd1f1db880b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:11:12.090445 20897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9b5a44a22ae454bbc45bd1f1db880b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:11:12.091641 20897 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9b5a44a22ae454bbc45bd1f1db880b3 [term 1 LEADER]: Becoming Leader. State: Replica: f9b5a44a22ae454bbc45bd1f1db880b3, State: Running, Role: LEADER
I20260926 03:11:12.092657 20897 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9b5a44a22ae454bbc45bd1f1db880b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b5a44a22ae454bbc45bd1f1db880b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37237 } }
I20260926 03:11:12.092871 20890 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9b5a44a22ae454bbc45bd1f1db880b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:11:12.096047 20900 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b5a44a22ae454bbc45bd1f1db880b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9b5a44a22ae454bbc45bd1f1db880b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b5a44a22ae454bbc45bd1f1db880b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37237 } } }
I20260926 03:11:12.096503 20900 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b5a44a22ae454bbc45bd1f1db880b3 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:12.097066 20901 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b5a44a22ae454bbc45bd1f1db880b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9b5a44a22ae454bbc45bd1f1db880b3. Latest consensus state: current_term: 1 leader_uuid: "f9b5a44a22ae454bbc45bd1f1db880b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b5a44a22ae454bbc45bd1f1db880b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37237 } } }
I20260926 03:11:12.097373 20901 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b5a44a22ae454bbc45bd1f1db880b3 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:12.105501 20907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:11:12.109000 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:12.092170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57837 (local address 127.0.0.1:37237)
0926 03:11:12.092786 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:12.092802 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:12.092828 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:11:12.093009 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:12.093014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:12.093090 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:12.093187 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:12.094903 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:12.096103 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:12.097907 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:12.098454 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:12.098576 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:12.101415 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:12.101421 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:12.101424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:12.101462 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:12.104797 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:12.105848 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:12.105854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:12.105857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:12.106003 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:12.106656 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:12.106662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:12.106665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:12.107106 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:12.107365 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:12.108367 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0926 03:11:12.108578 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482}
I20260926 03:11:12.112596 20907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:11:12.125586 20907 catalog_manager.cc:1347] Generated new cluster ID: 4d1638ec04c64ab68eb7d94b3779fe05
I20260926 03:11:12.125737 20907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:11:12.144068 20907 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:11:12.146072 20907 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:11:12.157546 20907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9b5a44a22ae454bbc45bd1f1db880b3: Generated new TSK 0
I20260926 03:11:12.158669 20907 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:11:12.173336 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 20823
I20260926 03:11:12.174613 20885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:11:12.267995 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 20823
W20260926 03:11:12.283218 20808 connection.cc:535] client connection to 127.0.0.1:37237 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37237): Connection reset by peer (error 104)
2026-09-26T03:11:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1947 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:11:12 dist-test-slave-hmbw krb5kdc[20922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:11:12 dist-test-slave-hmbw krb5kdc[20922](info): set up 2 sockets
Sep 26 03:11:12 dist-test-slave-hmbw krb5kdc[20922](info): commencing operation
krb5kdc: starting...
W20260926 03:11:13.302697 388 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:11:13 dist-test-slave-hmbw krb5kdc[20922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:11:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:11:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:11:13.554102 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44055
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:11:13.895653 20935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:13.895972 20935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:13.896062 20935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:13.903904 20935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:11:13.904047 20935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:13.904103 20935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:13.904166 20935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:11:13.904210 20935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:11:13.914721 20935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44055
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:13.916036 20935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:13.917372 20935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:13.929339 20941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:13.929708 20940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:13.929454 20943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:13.930212 20935 server_base.cc:1034] running on GCE node
I20260926 03:11:13.931062 20935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:13.932572 20935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:13.933794 20935 hybrid_clock.cc:648] HybridClock initialized: now 1790392273933739 us; error 47 us; skew 500 ppm
I20260926 03:11:13.934154 20935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:11:13 dist-test-slave-hmbw krb5kdc[20922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:11:13.937498 20935 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:11:13.939086 20935 webserver.cc:460] Webserver started at http://127.0.0.1:33233/ using document root <none> and password file <none>
I20260926 03:11:13.939776 20935 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:13.939889 20935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:13.940181 20935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:13.943157 20935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "4c96c192effd48ea9ab40362dc35b27a"
format_stamp: "Formatted at 2026-09-26 03:11:13 on dist-test-slave-hmbw"
server_key: "65cf499df41088b4c73a5bb44b8a42f2"
server_key_iv: "fed2f2462775b9bb52332202c90d55b4"
server_key_version: "encryptionkey@0"
I20260926 03:11:13.943924 20935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "4c96c192effd48ea9ab40362dc35b27a"
format_stamp: "Formatted at 2026-09-26 03:11:13 on dist-test-slave-hmbw"
server_key: "65cf499df41088b4c73a5bb44b8a42f2"
server_key_iv: "fed2f2462775b9bb52332202c90d55b4"
server_key_version: "encryptionkey@0"
I20260926 03:11:13.949224 20935 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:11:13.952780 20950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:13.954164 20935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:11:13.954339 20935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "4c96c192effd48ea9ab40362dc35b27a"
format_stamp: "Formatted at 2026-09-26 03:11:13 on dist-test-slave-hmbw"
server_key: "65cf499df41088b4c73a5bb44b8a42f2"
server_key_iv: "fed2f2462775b9bb52332202c90d55b4"
server_key_version: "encryptionkey@0"
I20260926 03:11:13.954605 20935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:13.970717 20935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:13.974910 20935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:13.975178 20935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:13.992820 20935 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44055
I20260926 03:11:13.992848 21001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44055 every 8 connection(s)
I20260926 03:11:13.994134 20935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:11:13.997896 21002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:13.998838 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 20935
I20260926 03:11:13.999006 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:11:13.999395 388 external_mini_cluster.cc:1428] Setting key 4fe563b7de3aa29eed10719e61a068d8
I20260926 03:11:14.013060 21002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c96c192effd48ea9ab40362dc35b27a: Bootstrap starting.
I20260926 03:11:14.015830 21005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:14.001308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43385 (local address 127.0.0.1:44055)
0926 03:11:14.002118 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:14.002137 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:14.002190 (+ 53us) server_negotiation.cc:407] Connection header received
0926 03:11:14.004444 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:14.004480 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:14.005104 (+ 624us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:14.005616 (+ 512us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:14.006144 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:14.008067 (+ 1923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:14.009755 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:14.011040 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:14.011296 (+ 256us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:14.012696 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:14.012708 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:14.012732 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:14.013500 (+ 768us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:14.013616 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:14.013658 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:14.013830 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:14.014202 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 03:11:14.014423 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":307,"threads_started":1}
I20260926 03:11:14.017446 21002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c96c192effd48ea9ab40362dc35b27a: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:14.018844 21002 log.cc:824] T 00000000000000000000000000000000 P 4c96c192effd48ea9ab40362dc35b27a: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:14.021657 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:11:14.023540 21002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c96c192effd48ea9ab40362dc35b27a: No bootstrap required, opened a new log
I20260926 03:11:14.033973 21002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c96c192effd48ea9ab40362dc35b27a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c96c192effd48ea9ab40362dc35b27a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44055 } }
I20260926 03:11:14.034454 21005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:14.026993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48201 (local address 127.0.0.1:44055)
0926 03:11:14.027216 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:14.027222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:14.027468 (+ 246us) server_negotiation.cc:407] Connection header received
0926 03:11:14.027618 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:14.027624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:14.027718 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:14.027858 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:14.028352 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:14.029508 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:14.031118 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:14.031814 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:14.031974 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:14.033548 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:14.033555 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:14.033559 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:14.033585 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:14.033659 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:14.033680 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:14.033805 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:14.034069 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 03:11:14.034128 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 03:11:14.034611 21002 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c96c192effd48ea9ab40362dc35b27a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:14.034782 21002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c96c192effd48ea9ab40362dc35b27a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c96c192effd48ea9ab40362dc35b27a, State: Initialized, Role: FOLLOWER
I20260926 03:11:14.035473 21002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c96c192effd48ea9ab40362dc35b27a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c96c192effd48ea9ab40362dc35b27a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44055 } }
I20260926 03:11:14.035799 21002 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c96c192effd48ea9ab40362dc35b27a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:11:14.035959 21002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c96c192effd48ea9ab40362dc35b27a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:11:14.036146 21002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c96c192effd48ea9ab40362dc35b27a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:14.038686 21002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c96c192effd48ea9ab40362dc35b27a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c96c192effd48ea9ab40362dc35b27a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44055 } }
I20260926 03:11:14.039283 21002 leader_election.cc:302] T 00000000000000000000000000000000 P 4c96c192effd48ea9ab40362dc35b27a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c96c192effd48ea9ab40362dc35b27a; no voters:
I20260926 03:11:14.039965 21002 leader_election.cc:288] T 00000000000000000000000000000000 P 4c96c192effd48ea9ab40362dc35b27a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:11:14.040097 21012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c96c192effd48ea9ab40362dc35b27a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:11:14.040794 21012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c96c192effd48ea9ab40362dc35b27a [term 1 LEADER]: Becoming Leader. State: Replica: 4c96c192effd48ea9ab40362dc35b27a, State: Running, Role: LEADER
I20260926 03:11:14.041388 21012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c96c192effd48ea9ab40362dc35b27a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c96c192effd48ea9ab40362dc35b27a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44055 } }
I20260926 03:11:14.042044 21002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c96c192effd48ea9ab40362dc35b27a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:11:14.044677 21013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c96c192effd48ea9ab40362dc35b27a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c96c192effd48ea9ab40362dc35b27a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c96c192effd48ea9ab40362dc35b27a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44055 } } }
I20260926 03:11:14.044728 21014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c96c192effd48ea9ab40362dc35b27a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c96c192effd48ea9ab40362dc35b27a. Latest consensus state: current_term: 1 leader_uuid: "4c96c192effd48ea9ab40362dc35b27a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c96c192effd48ea9ab40362dc35b27a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44055 } } }
I20260926 03:11:14.045192 21014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c96c192effd48ea9ab40362dc35b27a [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:14.045187 21013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c96c192effd48ea9ab40362dc35b27a [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:14.047659 21020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:11:14.053447 21020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:11:14.063818 21020 catalog_manager.cc:1347] Generated new cluster ID: 5245a89ddd8b4f9c8462c172fe8d7a55
I20260926 03:11:14.063959 21020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:11:14.092079 21020 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:11:14.093707 21020 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:11:14.102957 21020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c96c192effd48ea9ab40362dc35b27a: Generated new TSK 0
I20260926 03:11:14.103878 21020 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:11:14.123613 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 20935
W20260926 03:11:14.124966 20997 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43385
W20260926 03:11:14.125777 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 20935: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:11:14.125948 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 20935
W20260926 03:11:14.140020 20920 connection.cc:535] client connection to 127.0.0.1:44055 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44055): Connection reset by peer (error 104)
2026-09-26T03:11:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1855 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:11:14 dist-test-slave-hmbw krb5kdc[21033](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:11:14 dist-test-slave-hmbw krb5kdc[21033](info): set up 2 sockets
Sep 26 03:11:14 dist-test-slave-hmbw krb5kdc[21033](info): commencing operation
krb5kdc: starting...
W20260926 03:11:15.146690 388 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:11:15 dist-test-slave-hmbw krb5kdc[21033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:11:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:11:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:11:15.394862 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44041
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:11:15.724591 21046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:15.724912 21046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:15.725004 21046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:15.732816 21046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:11:15.732955 21046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:15.733003 21046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:15.733067 21046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:11:15.733109 21046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:11:15.743666 21046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44041
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:15.744998 21046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:15.746312 21046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:15.758209 21051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:15.758227 21052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:15.758402 21054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:15.759056 21046 server_base.cc:1034] running on GCE node
I20260926 03:11:15.759815 21046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:15.761257 21046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:15.762472 21046 hybrid_clock.cc:648] HybridClock initialized: now 1790392275762423 us; error 51 us; skew 500 ppm
I20260926 03:11:15.762849 21046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:11:15 dist-test-slave-hmbw krb5kdc[21033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:11:15.773487 21046 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:11:15.775245 21046 webserver.cc:460] Webserver started at http://127.0.0.1:42137/ using document root <none> and password file <none>
I20260926 03:11:15.775960 21046 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:15.776087 21046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:15.776384 21046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:15.779636 21046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "186e05fb59c14ef796e00da4ac234cdc"
format_stamp: "Formatted at 2026-09-26 03:11:15 on dist-test-slave-hmbw"
server_key: "56fdcad89276b494974a50e752b045bd"
server_key_iv: "17b53763ec5d2c49ffc975d07e974d62"
server_key_version: "encryptionkey@0"
I20260926 03:11:15.780447 21046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "186e05fb59c14ef796e00da4ac234cdc"
format_stamp: "Formatted at 2026-09-26 03:11:15 on dist-test-slave-hmbw"
server_key: "56fdcad89276b494974a50e752b045bd"
server_key_iv: "17b53763ec5d2c49ffc975d07e974d62"
server_key_version: "encryptionkey@0"
I20260926 03:11:15.785815 21046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 03:11:15.789535 21061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:15.791087 21046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:11:15.791273 21046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "186e05fb59c14ef796e00da4ac234cdc"
format_stamp: "Formatted at 2026-09-26 03:11:15 on dist-test-slave-hmbw"
server_key: "56fdcad89276b494974a50e752b045bd"
server_key_iv: "17b53763ec5d2c49ffc975d07e974d62"
server_key_version: "encryptionkey@0"
I20260926 03:11:15.791481 21046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:15.797561 21046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:15.802210 21046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:15.802543 21046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:15.822763 21046 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44041
I20260926 03:11:15.822779 21112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44041 every 8 connection(s)
I20260926 03:11:15.824385 21046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:11:15.828392 21113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:15.828866 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 21046
I20260926 03:11:15.829017 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:11:15.829473 388 external_mini_cluster.cc:1428] Setting key 7cd7e0f2b85c9ebebd607acd789a6f97
I20260926 03:11:15.838197 21116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:15.831311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53271 (local address 127.0.0.1:44041)
0926 03:11:15.832150 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:15.832162 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:15.832204 (+ 42us) server_negotiation.cc:407] Connection header received
0926 03:11:15.834721 (+ 2517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:15.834751 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:15.835188 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:15.835611 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:15.836617 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:15.836624 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:15.836637 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:15.836653 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:15.836718 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:15.836738 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:15.836954 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:15.837091 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 03:11:15.837270 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":377,"threads_started":1}
I20260926 03:11:15.842127 21113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 186e05fb59c14ef796e00da4ac234cdc: Bootstrap starting.
I20260926 03:11:15.842226 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:11:15.845386 21113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 186e05fb59c14ef796e00da4ac234cdc: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:15.846580 21113 log.cc:824] T 00000000000000000000000000000000 P 186e05fb59c14ef796e00da4ac234cdc: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:15.848827 21116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:15.846576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43913 (local address 127.0.0.1:44041)
0926 03:11:15.846754 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:15.846760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:15.846965 (+ 205us) server_negotiation.cc:407] Connection header received
0926 03:11:15.847140 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:15.847146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:15.847219 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:15.847333 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:15.848183 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:15.848188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:15.848192 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:15.848209 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:15.848256 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:15.848275 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:15.848386 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:15.848493 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 03:11:15.848556 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 03:11:15.849898 21113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 186e05fb59c14ef796e00da4ac234cdc: No bootstrap required, opened a new log
I20260926 03:11:15.857661 21113 raft_consensus.cc:348] T 00000000000000000000000000000000 P 186e05fb59c14ef796e00da4ac234cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186e05fb59c14ef796e00da4ac234cdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44041 } }
I20260926 03:11:15.858117 21113 raft_consensus.cc:374] T 00000000000000000000000000000000 P 186e05fb59c14ef796e00da4ac234cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:15.858212 21113 raft_consensus.cc:729] T 00000000000000000000000000000000 P 186e05fb59c14ef796e00da4ac234cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186e05fb59c14ef796e00da4ac234cdc, State: Initialized, Role: FOLLOWER
I20260926 03:11:15.858899 21113 consensus_queue.cc:260] T 00000000000000000000000000000000 P 186e05fb59c14ef796e00da4ac234cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186e05fb59c14ef796e00da4ac234cdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44041 } }
I20260926 03:11:15.859195 21113 raft_consensus.cc:388] T 00000000000000000000000000000000 P 186e05fb59c14ef796e00da4ac234cdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:11:15.859334 21113 raft_consensus.cc:482] T 00000000000000000000000000000000 P 186e05fb59c14ef796e00da4ac234cdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:11:15.859490 21113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 186e05fb59c14ef796e00da4ac234cdc [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:15.861671 21113 raft_consensus.cc:504] T 00000000000000000000000000000000 P 186e05fb59c14ef796e00da4ac234cdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186e05fb59c14ef796e00da4ac234cdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44041 } }
I20260926 03:11:15.862215 21113 leader_election.cc:302] T 00000000000000000000000000000000 P 186e05fb59c14ef796e00da4ac234cdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 186e05fb59c14ef796e00da4ac234cdc; no voters:
I20260926 03:11:15.862875 21113 leader_election.cc:288] T 00000000000000000000000000000000 P 186e05fb59c14ef796e00da4ac234cdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:11:15.863024 21123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 186e05fb59c14ef796e00da4ac234cdc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:11:15.863678 21123 raft_consensus.cc:686] T 00000000000000000000000000000000 P 186e05fb59c14ef796e00da4ac234cdc [term 1 LEADER]: Becoming Leader. State: Replica: 186e05fb59c14ef796e00da4ac234cdc, State: Running, Role: LEADER
I20260926 03:11:15.864447 21123 consensus_queue.cc:237] T 00000000000000000000000000000000 P 186e05fb59c14ef796e00da4ac234cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186e05fb59c14ef796e00da4ac234cdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44041 } }
I20260926 03:11:15.865056 21113 sys_catalog.cc:564] T 00000000000000000000000000000000 P 186e05fb59c14ef796e00da4ac234cdc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:11:15.867115 21124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 186e05fb59c14ef796e00da4ac234cdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "186e05fb59c14ef796e00da4ac234cdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186e05fb59c14ef796e00da4ac234cdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44041 } } }
I20260926 03:11:15.867547 21124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 186e05fb59c14ef796e00da4ac234cdc [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:15.868022 21125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 186e05fb59c14ef796e00da4ac234cdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 186e05fb59c14ef796e00da4ac234cdc. Latest consensus state: current_term: 1 leader_uuid: "186e05fb59c14ef796e00da4ac234cdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186e05fb59c14ef796e00da4ac234cdc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44041 } } }
I20260926 03:11:15.868316 21125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 186e05fb59c14ef796e00da4ac234cdc [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:15.868558 21129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:11:15.874694 21129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:11:15.883831 21129 catalog_manager.cc:1347] Generated new cluster ID: c47410ee7766402092017f799b47d3e4
I20260926 03:11:15.883970 21129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:11:15.899367 21129 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:11:15.899896 21129 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:11:15.921188 21129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 186e05fb59c14ef796e00da4ac234cdc: Generated new TSK 0
I20260926 03:11:15.922320 21129 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:11:15.936165 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 21046
W20260926 03:11:15.937227 21108 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53271
W20260926 03:11:15.937924 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 21046: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:11:15.938045 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 21046
2026-09-26T03:11:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1811 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:11:16 dist-test-slave-hmbw krb5kdc[21144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:11:16 dist-test-slave-hmbw krb5kdc[21144](info): set up 2 sockets
Sep 26 03:11:16 dist-test-slave-hmbw krb5kdc[21144](info): commencing operation
krb5kdc: starting...
W20260926 03:11:16.962626 388 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:11:17 dist-test-slave-hmbw krb5kdc[21144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:11:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:11:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:11:17.214006 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37353
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:11:17.537988 21157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:17.538298 21157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:17.538419 21157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:17.545835 21157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:11:17.545953 21157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:17.546011 21157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:17.546089 21157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:11:17.546146 21157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:11:17.556095 21157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:17.557410 21157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:17.558686 21157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:17.570139 21162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:17.570104 21165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:17.570070 21163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:17.570766 21157 server_base.cc:1034] running on GCE node
I20260926 03:11:17.571551 21157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:17.572994 21157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:17.574229 21157 hybrid_clock.cc:648] HybridClock initialized: now 1790392277574173 us; error 32 us; skew 500 ppm
I20260926 03:11:17.574656 21157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:11:17 dist-test-slave-hmbw krb5kdc[21144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:11:17.577630 21157 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:11:17.579139 21157 webserver.cc:460] Webserver started at http://127.0.0.1:39241/ using document root <none> and password file <none>
I20260926 03:11:17.579803 21157 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:17.579917 21157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:17.580195 21157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:17.583166 21157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "fe2a5e79161f4e669b5fe794e23eceb7"
format_stamp: "Formatted at 2026-09-26 03:11:17 on dist-test-slave-hmbw"
server_key: "986c30fdd39c70f5e978b5453502762b"
server_key_iv: "114b0fb6abb63b7484e512ac1bfb94d6"
server_key_version: "encryptionkey@0"
I20260926 03:11:17.583920 21157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "fe2a5e79161f4e669b5fe794e23eceb7"
format_stamp: "Formatted at 2026-09-26 03:11:17 on dist-test-slave-hmbw"
server_key: "986c30fdd39c70f5e978b5453502762b"
server_key_iv: "114b0fb6abb63b7484e512ac1bfb94d6"
server_key_version: "encryptionkey@0"
I20260926 03:11:17.588966 21157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 03:11:17.593129 21172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:17.595059 21157 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 03:11:17.595288 21157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "fe2a5e79161f4e669b5fe794e23eceb7"
format_stamp: "Formatted at 2026-09-26 03:11:17 on dist-test-slave-hmbw"
server_key: "986c30fdd39c70f5e978b5453502762b"
server_key_iv: "114b0fb6abb63b7484e512ac1bfb94d6"
server_key_version: "encryptionkey@0"
I20260926 03:11:17.595546 21157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:17.628583 21157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:17.633121 21157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:17.633375 21157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:17.651265 21157 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37353
I20260926 03:11:17.651288 21223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37353 every 8 connection(s)
I20260926 03:11:17.652581 21157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:11:17.656396 21224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:17.657825 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 21157
I20260926 03:11:17.657996 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:11:17.658419 388 external_mini_cluster.cc:1428] Setting key b2461ad7f9b65adfc3529f6f1f285c01
I20260926 03:11:17.670312 21224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe2a5e79161f4e669b5fe794e23eceb7: Bootstrap starting.
Sep 26 03:11:17 dist-test-slave-hmbw krb5kdc[21144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 03:11:17.674340 21224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe2a5e79161f4e669b5fe794e23eceb7: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:17.675410 21224 log.cc:824] T 00000000000000000000000000000000 P fe2a5e79161f4e669b5fe794e23eceb7: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:17.678539 21224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe2a5e79161f4e669b5fe794e23eceb7: No bootstrap required, opened a new log
I20260926 03:11:17.681998 21227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:17.660292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43583 (local address 127.0.0.1:37353)
0926 03:11:17.661093 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:17.661109 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:17.661153 (+ 44us) server_negotiation.cc:407] Connection header received
0926 03:11:17.662761 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:17.662792 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:17.663252 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:17.663600 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:17.665278 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:17.666649 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:17.668516 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:17.669252 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:17.669395 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:17.674311 (+ 4916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:17.674320 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:17.674340 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:17.674388 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:17.677654 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:17.678589 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:17.678598 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:17.678606 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:17.678791 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:17.679249 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:17.679253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:17.679255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:17.679793 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:17.680029 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:17.680493 (+ 464us) server_negotiation.cc:299] Negotiation successful
0926 03:11:17.680874 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":350,"threads_started":1}
I20260926 03:11:17.685737 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:11:17.687464 21224 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe2a5e79161f4e669b5fe794e23eceb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2a5e79161f4e669b5fe794e23eceb7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37353 } }
I20260926 03:11:17.687922 21224 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe2a5e79161f4e669b5fe794e23eceb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:17.688020 21224 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe2a5e79161f4e669b5fe794e23eceb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe2a5e79161f4e669b5fe794e23eceb7, State: Initialized, Role: FOLLOWER
I20260926 03:11:17.688685 21224 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe2a5e79161f4e669b5fe794e23eceb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2a5e79161f4e669b5fe794e23eceb7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37353 } }
I20260926 03:11:17.688992 21224 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe2a5e79161f4e669b5fe794e23eceb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:11:17.689139 21224 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe2a5e79161f4e669b5fe794e23eceb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:11:17.689318 21224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe2a5e79161f4e669b5fe794e23eceb7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:17.692200 21224 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe2a5e79161f4e669b5fe794e23eceb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2a5e79161f4e669b5fe794e23eceb7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37353 } }
I20260926 03:11:17.692742 21224 leader_election.cc:302] T 00000000000000000000000000000000 P fe2a5e79161f4e669b5fe794e23eceb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe2a5e79161f4e669b5fe794e23eceb7; no voters:
I20260926 03:11:17.693332 21224 leader_election.cc:288] T 00000000000000000000000000000000 P fe2a5e79161f4e669b5fe794e23eceb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:11:17.693444 21234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe2a5e79161f4e669b5fe794e23eceb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:11:17.694630 21234 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe2a5e79161f4e669b5fe794e23eceb7 [term 1 LEADER]: Becoming Leader. State: Replica: fe2a5e79161f4e669b5fe794e23eceb7, State: Running, Role: LEADER
I20260926 03:11:17.695189 21234 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe2a5e79161f4e669b5fe794e23eceb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2a5e79161f4e669b5fe794e23eceb7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37353 } }
I20260926 03:11:17.695695 21224 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe2a5e79161f4e669b5fe794e23eceb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:11:17.698002 21235 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe2a5e79161f4e669b5fe794e23eceb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe2a5e79161f4e669b5fe794e23eceb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2a5e79161f4e669b5fe794e23eceb7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37353 } } }
I20260926 03:11:17.698367 21235 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe2a5e79161f4e669b5fe794e23eceb7 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:17.701385 21239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:11:17.701016 21236 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe2a5e79161f4e669b5fe794e23eceb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe2a5e79161f4e669b5fe794e23eceb7. Latest consensus state: current_term: 1 leader_uuid: "fe2a5e79161f4e669b5fe794e23eceb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2a5e79161f4e669b5fe794e23eceb7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37353 } } }
I20260926 03:11:17.701853 21236 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe2a5e79161f4e669b5fe794e23eceb7 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:17.707266 21239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:11:17.708941 21227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:17.690847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57973 (local address 127.0.0.1:37353)
0926 03:11:17.691036 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:17.691042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:17.691358 (+ 316us) server_negotiation.cc:407] Connection header received
0926 03:11:17.691484 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:17.691489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:17.691564 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:17.691667 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:17.693093 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:17.694285 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:17.696202 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:17.696940 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:17.697075 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:17.700571 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:17.700578 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:17.700582 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:17.700627 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:17.704554 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:17.705562 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:17.705568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:17.705571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:17.705732 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:17.706240 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:17.706244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:17.706247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:17.706684 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:17.708130 (+ 1446us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:17.708277 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 03:11:17.708518 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:11:17.718888 21239 catalog_manager.cc:1347] Generated new cluster ID: 96321e6494a54c5887d71ebf1e6c629a
I20260926 03:11:17.719015 21239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:11:17.739851 21239 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:11:17.741154 21239 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:11:17.747401 21239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe2a5e79161f4e669b5fe794e23eceb7: Generated new TSK 0
I20260926 03:11:17.748224 21239 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:11:17.760867 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 21157
I20260926 03:11:17.762715 21219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:11:17.860991 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 21157
2026-09-26T03:11:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1923 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:11:17 dist-test-slave-hmbw krb5kdc[21256](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:11:17 dist-test-slave-hmbw krb5kdc[21256](info): set up 2 sockets
Sep 26 03:11:17 dist-test-slave-hmbw krb5kdc[21256](info): commencing operation
krb5kdc: starting...
W20260926 03:11:18.874737 388 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:11:19 dist-test-slave-hmbw krb5kdc[21256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:11:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:11:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:11:19.130712 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35485
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:11:19.459426 21269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:19.459731 21269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:19.459821 21269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:19.467144 21269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:11:19.467273 21269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:19.467317 21269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:19.467371 21269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:11:19.467422 21269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:11:19.477394 21269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:19.478926 21269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:19.480245 21269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:19.493923 21274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:19.493970 21275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:19.493993 21277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:19.494933 21269 server_base.cc:1034] running on GCE node
I20260926 03:11:19.495735 21269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:19.497145 21269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:19.498370 21269 hybrid_clock.cc:648] HybridClock initialized: now 1790392279498317 us; error 48 us; skew 500 ppm
I20260926 03:11:19.498730 21269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:11:19 dist-test-slave-hmbw krb5kdc[21256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:11:19.501888 21269 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:11:19.503611 21269 webserver.cc:460] Webserver started at http://127.0.0.1:35411/ using document root <none> and password file <none>
I20260926 03:11:19.504312 21269 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:19.504412 21269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:19.504667 21269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:19.507691 21269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "ca7d570134b54854beebb9831c66af37"
format_stamp: "Formatted at 2026-09-26 03:11:19 on dist-test-slave-hmbw"
server_key: "00a2fafdc598b85f85b9e90e2fb1783e"
server_key_iv: "d73970aa55dd9f1ca29f4c82a817db95"
server_key_version: "encryptionkey@0"
I20260926 03:11:19.508441 21269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "ca7d570134b54854beebb9831c66af37"
format_stamp: "Formatted at 2026-09-26 03:11:19 on dist-test-slave-hmbw"
server_key: "00a2fafdc598b85f85b9e90e2fb1783e"
server_key_iv: "d73970aa55dd9f1ca29f4c82a817db95"
server_key_version: "encryptionkey@0"
I20260926 03:11:19.513800 21269 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:11:19.517498 21284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:19.518989 21269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 03:11:19.519155 21269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "ca7d570134b54854beebb9831c66af37"
format_stamp: "Formatted at 2026-09-26 03:11:19 on dist-test-slave-hmbw"
server_key: "00a2fafdc598b85f85b9e90e2fb1783e"
server_key_iv: "d73970aa55dd9f1ca29f4c82a817db95"
server_key_version: "encryptionkey@0"
I20260926 03:11:19.519354 21269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:19.539805 21269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:19.544149 21269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:19.544413 21269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:19.561962 21269 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35485
I20260926 03:11:19.561970 21335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35485 every 8 connection(s)
I20260926 03:11:19.563267 21269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:11:19.565083 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 21269
I20260926 03:11:19.565232 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:11:19.565716 388 external_mini_cluster.cc:1428] Setting key 2a88d0d7efb29275af93c324059b5214
I20260926 03:11:19.567521 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:19.580452 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:19.567489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45939 (local address 127.0.0.1:35485)
0926 03:11:19.568273 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:19.568289 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:19.568339 (+ 50us) server_negotiation.cc:407] Connection header received
0926 03:11:19.570448 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:19.570477 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:19.571016 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:19.571473 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:19.571970 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:19.573947 (+ 1977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:19.575366 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:19.576377 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:19.576548 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:19.577636 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:19.577650 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:19.577669 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:19.578412 (+ 743us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:19.578529 (+ 117us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:19.578571 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:19.578683 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:19.578909 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 03:11:19.579192 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":307,"threads_started":1}
I20260926 03:11:19.581579 21336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca7d570134b54854beebb9831c66af37: Bootstrap starting.
I20260926 03:11:19.584022 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:11:19.584731 21336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca7d570134b54854beebb9831c66af37: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:19.585686 21336 log.cc:824] T 00000000000000000000000000000000 P ca7d570134b54854beebb9831c66af37: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:19.588976 21336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca7d570134b54854beebb9831c66af37: No bootstrap required, opened a new log
I20260926 03:11:19.593950 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:19.588236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46351 (local address 127.0.0.1:35485)
0926 03:11:19.588389 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:19.588395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:19.588754 (+ 359us) server_negotiation.cc:407] Connection header received
0926 03:11:19.588931 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:19.588937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:19.589009 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:19.589089 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:19.589607 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:19.590344 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:19.591685 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:19.592139 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:19.592254 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:19.593241 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:19.593246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:19.593249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:19.593267 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:19.593310 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:19.593329 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:19.593407 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:19.593536 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 03:11:19.593635 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 03:11:19.596840 21336 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca7d570134b54854beebb9831c66af37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7d570134b54854beebb9831c66af37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35485 } }
I20260926 03:11:19.597219 21336 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca7d570134b54854beebb9831c66af37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:19.597308 21336 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca7d570134b54854beebb9831c66af37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca7d570134b54854beebb9831c66af37, State: Initialized, Role: FOLLOWER
I20260926 03:11:19.597836 21336 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca7d570134b54854beebb9831c66af37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7d570134b54854beebb9831c66af37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35485 } }
I20260926 03:11:19.598053 21336 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca7d570134b54854beebb9831c66af37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:11:19.598174 21336 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca7d570134b54854beebb9831c66af37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:11:19.598333 21336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca7d570134b54854beebb9831c66af37 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:19.600759 21336 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca7d570134b54854beebb9831c66af37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7d570134b54854beebb9831c66af37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35485 } }
I20260926 03:11:19.601254 21336 leader_election.cc:302] T 00000000000000000000000000000000 P ca7d570134b54854beebb9831c66af37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca7d570134b54854beebb9831c66af37; no voters:
I20260926 03:11:19.601778 21336 leader_election.cc:288] T 00000000000000000000000000000000 P ca7d570134b54854beebb9831c66af37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:11:19.601910 21346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca7d570134b54854beebb9831c66af37 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:11:19.602655 21346 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca7d570134b54854beebb9831c66af37 [term 1 LEADER]: Becoming Leader. State: Replica: ca7d570134b54854beebb9831c66af37, State: Running, Role: LEADER
I20260926 03:11:19.603227 21346 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca7d570134b54854beebb9831c66af37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7d570134b54854beebb9831c66af37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35485 } }
I20260926 03:11:19.603739 21336 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca7d570134b54854beebb9831c66af37 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:11:19.605784 21348 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca7d570134b54854beebb9831c66af37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca7d570134b54854beebb9831c66af37. Latest consensus state: current_term: 1 leader_uuid: "ca7d570134b54854beebb9831c66af37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7d570134b54854beebb9831c66af37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35485 } } }
I20260926 03:11:19.606182 21348 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca7d570134b54854beebb9831c66af37 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:19.606792 21347 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca7d570134b54854beebb9831c66af37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca7d570134b54854beebb9831c66af37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7d570134b54854beebb9831c66af37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35485 } } }
I20260926 03:11:19.607084 21347 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca7d570134b54854beebb9831c66af37 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:19.608570 21351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:11:19.613945 21351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:11:19.624788 21351 catalog_manager.cc:1347] Generated new cluster ID: c7d7b24244cb40d5b97e5d47d839925d
I20260926 03:11:19.624917 21351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:11:19.645237 21351 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:11:19.647341 21351 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:11:19.673997 21351 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca7d570134b54854beebb9831c66af37: Generated new TSK 0
I20260926 03:11:19.675148 21351 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:11:19.691388 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 21269
W20260926 03:11:19.693030 21331 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45939
W20260926 03:11:19.693778 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 21269: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:11:19.693923 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 21269
2026-09-26T03:11:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1832 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:11:19 dist-test-slave-hmbw krb5kdc[21367](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:11:19 dist-test-slave-hmbw krb5kdc[21367](info): set up 2 sockets
Sep 26 03:11:19 dist-test-slave-hmbw krb5kdc[21367](info): commencing operation
krb5kdc: starting...
W20260926 03:11:20.734686 388 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:11:20 dist-test-slave-hmbw krb5kdc[21367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:11:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:11:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:11:20.986913 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33789
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:11:21.324929 21380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:21.325251 21380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:21.325357 21380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:21.332870 21380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:11:21.333004 21380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:21.333050 21380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:21.333117 21380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:11:21.333160 21380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:11:21.343291 21380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:21.344530 21380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:21.345793 21380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:21.357590 21385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:21.357542 21386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:21.357522 21388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:21.358389 21380 server_base.cc:1034] running on GCE node
I20260926 03:11:21.359206 21380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:21.360743 21380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:21.361989 21380 hybrid_clock.cc:648] HybridClock initialized: now 1790392281361919 us; error 46 us; skew 500 ppm
I20260926 03:11:21.362361 21380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:11:21 dist-test-slave-hmbw krb5kdc[21367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:11:21.371699 21380 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:11:21.373564 21380 webserver.cc:460] Webserver started at http://127.0.0.1:40401/ using document root <none> and password file <none>
I20260926 03:11:21.374317 21380 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:21.374432 21380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:21.374766 21380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:21.377859 21380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "a3d56c6ebedc497ea9e0ddc636c09565"
format_stamp: "Formatted at 2026-09-26 03:11:21 on dist-test-slave-hmbw"
server_key: "0184f541f7cde9c6459d9e895f4c48e9"
server_key_iv: "dabcbd1efa8aedccd8f08f04aa2e09a7"
server_key_version: "encryptionkey@0"
I20260926 03:11:21.378662 21380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "a3d56c6ebedc497ea9e0ddc636c09565"
format_stamp: "Formatted at 2026-09-26 03:11:21 on dist-test-slave-hmbw"
server_key: "0184f541f7cde9c6459d9e895f4c48e9"
server_key_iv: "dabcbd1efa8aedccd8f08f04aa2e09a7"
server_key_version: "encryptionkey@0"
I20260926 03:11:21.384051 21380 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 03:11:21.387843 21395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:21.389333 21380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 03:11:21.389514 21380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "a3d56c6ebedc497ea9e0ddc636c09565"
format_stamp: "Formatted at 2026-09-26 03:11:21 on dist-test-slave-hmbw"
server_key: "0184f541f7cde9c6459d9e895f4c48e9"
server_key_iv: "dabcbd1efa8aedccd8f08f04aa2e09a7"
server_key_version: "encryptionkey@0"
I20260926 03:11:21.389750 21380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:21.395565 21380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:21.400075 21380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:21.400369 21380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:21.418045 21380 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33789
I20260926 03:11:21.418097 21446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33789 every 8 connection(s)
I20260926 03:11:21.419536 21380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:11:21.421440 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 21380
I20260926 03:11:21.421617 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:11:21.422144 388 external_mini_cluster.cc:1428] Setting key 2baedf6bdde7c3ec6fb7b4a3756662c3
I20260926 03:11:21.424221 21447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:21.430207 21450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:21.424037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57657 (local address 127.0.0.1:33789)
0926 03:11:21.424887 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:21.424900 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:21.424942 (+ 42us) server_negotiation.cc:407] Connection header received
0926 03:11:21.426702 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:21.426724 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:21.427106 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:21.427490 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:21.428539 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:21.428545 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:21.428558 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:21.428574 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:21.428655 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:21.428676 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:21.428906 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:21.429046 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 03:11:21.429238 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":351,"threads_started":1}
I20260926 03:11:21.434360 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:11:21.436471 21447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3d56c6ebedc497ea9e0ddc636c09565: Bootstrap starting.
I20260926 03:11:21.440157 21447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3d56c6ebedc497ea9e0ddc636c09565: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:21.441184 21450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:21.438721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54641 (local address 127.0.0.1:33789)
0926 03:11:21.438905 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:21.438911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:21.439133 (+ 222us) server_negotiation.cc:407] Connection header received
0926 03:11:21.439344 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:21.439348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:21.439432 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:21.439523 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:21.440563 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:21.440567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:21.440571 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:21.440587 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:21.440631 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:21.440651 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:21.440758 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:21.440867 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 03:11:21.440913 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:11:21.441529 21447 log.cc:824] T 00000000000000000000000000000000 P a3d56c6ebedc497ea9e0ddc636c09565: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:21.444469 21447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3d56c6ebedc497ea9e0ddc636c09565: No bootstrap required, opened a new log
I20260926 03:11:21.452276 21447 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3d56c6ebedc497ea9e0ddc636c09565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3d56c6ebedc497ea9e0ddc636c09565" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33789 } }
I20260926 03:11:21.452811 21447 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3d56c6ebedc497ea9e0ddc636c09565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:21.452924 21447 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3d56c6ebedc497ea9e0ddc636c09565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3d56c6ebedc497ea9e0ddc636c09565, State: Initialized, Role: FOLLOWER
I20260926 03:11:21.453621 21447 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3d56c6ebedc497ea9e0ddc636c09565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3d56c6ebedc497ea9e0ddc636c09565" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33789 } }
I20260926 03:11:21.453878 21447 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3d56c6ebedc497ea9e0ddc636c09565 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:11:21.453991 21447 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3d56c6ebedc497ea9e0ddc636c09565 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:11:21.454116 21447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3d56c6ebedc497ea9e0ddc636c09565 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:21.456400 21447 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3d56c6ebedc497ea9e0ddc636c09565 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3d56c6ebedc497ea9e0ddc636c09565" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33789 } }
I20260926 03:11:21.456913 21447 leader_election.cc:302] T 00000000000000000000000000000000 P a3d56c6ebedc497ea9e0ddc636c09565 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3d56c6ebedc497ea9e0ddc636c09565; no voters:
I20260926 03:11:21.457513 21447 leader_election.cc:288] T 00000000000000000000000000000000 P a3d56c6ebedc497ea9e0ddc636c09565 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:11:21.457679 21457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3d56c6ebedc497ea9e0ddc636c09565 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:11:21.458376 21457 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3d56c6ebedc497ea9e0ddc636c09565 [term 1 LEADER]: Becoming Leader. State: Replica: a3d56c6ebedc497ea9e0ddc636c09565, State: Running, Role: LEADER
I20260926 03:11:21.458958 21457 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3d56c6ebedc497ea9e0ddc636c09565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3d56c6ebedc497ea9e0ddc636c09565" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33789 } }
I20260926 03:11:21.459545 21447 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3d56c6ebedc497ea9e0ddc636c09565 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:11:21.462338 21459 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3d56c6ebedc497ea9e0ddc636c09565 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3d56c6ebedc497ea9e0ddc636c09565. Latest consensus state: current_term: 1 leader_uuid: "a3d56c6ebedc497ea9e0ddc636c09565" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3d56c6ebedc497ea9e0ddc636c09565" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33789 } } }
I20260926 03:11:21.462770 21459 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3d56c6ebedc497ea9e0ddc636c09565 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:21.463465 21458 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3d56c6ebedc497ea9e0ddc636c09565 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3d56c6ebedc497ea9e0ddc636c09565" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3d56c6ebedc497ea9e0ddc636c09565" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33789 } } }
I20260926 03:11:21.463846 21458 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3d56c6ebedc497ea9e0ddc636c09565 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:21.464267 21463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:11:21.470702 21463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:11:21.479813 21463 catalog_manager.cc:1347] Generated new cluster ID: e91fc2a8f70242f38a25e0e54d586ed3
I20260926 03:11:21.479943 21463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:11:21.494395 21463 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:11:21.494874 21463 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:11:21.522464 21463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3d56c6ebedc497ea9e0ddc636c09565: Generated new TSK 0
I20260926 03:11:21.523453 21463 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:11:21.542922 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 21380
W20260926 03:11:21.544872 21442 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57657
W20260926 03:11:21.545754 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 21380: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:11:21.545918 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 21380
2026-09-26T03:11:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1854 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:11:21 dist-test-slave-hmbw krb5kdc[21478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:11:21 dist-test-slave-hmbw krb5kdc[21478](info): set up 2 sockets
Sep 26 03:11:21 dist-test-slave-hmbw krb5kdc[21478](info): commencing operation
krb5kdc: starting...
W20260926 03:11:22.578722 388 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:11:22 dist-test-slave-hmbw krb5kdc[21478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:11:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:11:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:11:22.838430 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45611
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:11:23.173485 21491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:23.173786 21491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:23.173877 21491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:23.181466 21491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:11:23.181588 21491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:23.181635 21491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:23.181697 21491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:11:23.181739 21491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:11:23.191912 21491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:23.193200 21491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:23.194465 21491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:23.206472 21497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:23.206521 21499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:23.206420 21496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:23.207396 21491 server_base.cc:1034] running on GCE node
I20260926 03:11:23.208200 21491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:23.209673 21491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:23.210886 21491 hybrid_clock.cc:648] HybridClock initialized: now 1790392283210819 us; error 61 us; skew 500 ppm
I20260926 03:11:23.211259 21491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:11:23 dist-test-slave-hmbw krb5kdc[21478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:11:23.214584 21491 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:11:23.216219 21491 webserver.cc:460] Webserver started at http://127.0.0.1:37909/ using document root <none> and password file <none>
I20260926 03:11:23.216917 21491 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:23.217024 21491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:23.217327 21491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:23.220405 21491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "4f92c53ed27e4de8bd2acc723e587435"
format_stamp: "Formatted at 2026-09-26 03:11:23 on dist-test-slave-hmbw"
server_key: "4e520109973034e2cab2d86ea219682e"
server_key_iv: "56292465666f4220c698e57493bc5a11"
server_key_version: "encryptionkey@0"
I20260926 03:11:23.221170 21491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "4f92c53ed27e4de8bd2acc723e587435"
format_stamp: "Formatted at 2026-09-26 03:11:23 on dist-test-slave-hmbw"
server_key: "4e520109973034e2cab2d86ea219682e"
server_key_iv: "56292465666f4220c698e57493bc5a11"
server_key_version: "encryptionkey@0"
I20260926 03:11:23.226441 21491 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:11:23.229821 21506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:23.231200 21491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:11:23.231376 21491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "4f92c53ed27e4de8bd2acc723e587435"
format_stamp: "Formatted at 2026-09-26 03:11:23 on dist-test-slave-hmbw"
server_key: "4e520109973034e2cab2d86ea219682e"
server_key_iv: "56292465666f4220c698e57493bc5a11"
server_key_version: "encryptionkey@0"
I20260926 03:11:23.231575 21491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:23.250293 21491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:23.254961 21491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:23.255360 21491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:23.273372 21491 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45611
I20260926 03:11:23.273393 21557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45611 every 8 connection(s)
I20260926 03:11:23.274855 21491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:11:23.278877 21558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:23.282974 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 21491
I20260926 03:11:23.283146 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:11:23.283557 388 external_mini_cluster.cc:1428] Setting key 64782b23bd1a1ec8e098f24488334204
I20260926 03:11:23.291814 21558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f92c53ed27e4de8bd2acc723e587435: Bootstrap starting.
I20260926 03:11:23.295202 21558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f92c53ed27e4de8bd2acc723e587435: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:23.296236 21558 log.cc:824] T 00000000000000000000000000000000 P 4f92c53ed27e4de8bd2acc723e587435: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:11:23 dist-test-slave-hmbw krb5kdc[21478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 03:11:23.300560 21558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f92c53ed27e4de8bd2acc723e587435: No bootstrap required, opened a new log
I20260926 03:11:23.309688 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:23.285506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:34491 (local address 127.0.0.1:45611)
0926 03:11:23.286292 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:23.286304 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:23.286346 (+ 42us) server_negotiation.cc:407] Connection header received
0926 03:11:23.288086 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:23.288112 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:23.288581 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:23.288958 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:23.290425 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:23.291844 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:23.293150 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:23.293874 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:23.294046 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:23.298342 (+ 4296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:23.298357 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:23.298378 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:23.298442 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:23.303809 (+ 5367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:23.304987 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:23.304999 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:23.305010 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:23.305292 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:23.305881 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:23.305886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:23.305889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:23.306890 (+ 1001us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:23.307313 (+ 423us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:23.307747 (+ 434us) server_negotiation.cc:299] Negotiation successful
0926 03:11:23.308176 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":331,"threads_started":1}
I20260926 03:11:23.311880 21558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f92c53ed27e4de8bd2acc723e587435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f92c53ed27e4de8bd2acc723e587435" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45611 } }
I20260926 03:11:23.312366 21558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f92c53ed27e4de8bd2acc723e587435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:23.312495 21558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f92c53ed27e4de8bd2acc723e587435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f92c53ed27e4de8bd2acc723e587435, State: Initialized, Role: FOLLOWER
I20260926 03:11:23.313086 21558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f92c53ed27e4de8bd2acc723e587435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f92c53ed27e4de8bd2acc723e587435" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45611 } }
I20260926 03:11:23.313383 21558 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f92c53ed27e4de8bd2acc723e587435 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:11:23.313547 21558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f92c53ed27e4de8bd2acc723e587435 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:11:23.313732 21558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f92c53ed27e4de8bd2acc723e587435 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:23.314464 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:11:23.316185 21558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f92c53ed27e4de8bd2acc723e587435 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f92c53ed27e4de8bd2acc723e587435" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45611 } }
I20260926 03:11:23.316749 21558 leader_election.cc:302] T 00000000000000000000000000000000 P 4f92c53ed27e4de8bd2acc723e587435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f92c53ed27e4de8bd2acc723e587435; no voters:
I20260926 03:11:23.317363 21558 leader_election.cc:288] T 00000000000000000000000000000000 P 4f92c53ed27e4de8bd2acc723e587435 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:11:23.317554 21565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f92c53ed27e4de8bd2acc723e587435 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:11:23.318819 21565 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f92c53ed27e4de8bd2acc723e587435 [term 1 LEADER]: Becoming Leader. State: Replica: 4f92c53ed27e4de8bd2acc723e587435, State: Running, Role: LEADER
I20260926 03:11:23.319674 21565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f92c53ed27e4de8bd2acc723e587435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f92c53ed27e4de8bd2acc723e587435" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45611 } }
I20260926 03:11:23.321108 21558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f92c53ed27e4de8bd2acc723e587435 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:11:23.323504 21566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f92c53ed27e4de8bd2acc723e587435 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f92c53ed27e4de8bd2acc723e587435. Latest consensus state: current_term: 1 leader_uuid: "4f92c53ed27e4de8bd2acc723e587435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f92c53ed27e4de8bd2acc723e587435" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45611 } } }
I20260926 03:11:23.323951 21566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f92c53ed27e4de8bd2acc723e587435 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:23.324514 21567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f92c53ed27e4de8bd2acc723e587435 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f92c53ed27e4de8bd2acc723e587435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f92c53ed27e4de8bd2acc723e587435" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45611 } } }
I20260926 03:11:23.324849 21567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f92c53ed27e4de8bd2acc723e587435 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:23.324983 21574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:11:23.338727 21574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:11:23.350282 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:23.322944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:37817 (local address 127.0.0.1:45611)
0926 03:11:23.323152 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:23.323159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:23.331738 (+ 8579us) server_negotiation.cc:407] Connection header received
0926 03:11:23.332042 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:23.332048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:23.332146 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:23.332224 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:23.334330 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:23.335710 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:23.337775 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:23.338516 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:23.338637 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:23.341981 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:23.341988 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:23.341997 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:23.342047 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:23.346030 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:23.347204 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:23.347210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:23.347213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:23.347397 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:23.348068 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:23.348074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:23.348079 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:23.348533 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:23.348824 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:23.349604 (+ 780us) server_negotiation.cc:299] Negotiation successful
0926 03:11:23.349846 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:11:23.357223 21574 catalog_manager.cc:1347] Generated new cluster ID: ae4bce9f11d140a3a945069de5d03eb1
I20260926 03:11:23.357364 21574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:11:23.389904 21574 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:11:23.391600 21574 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:11:23.401768 21574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f92c53ed27e4de8bd2acc723e587435: Generated new TSK 0
I20260926 03:11:23.402776 21574 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:11:23.420158 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 21491
I20260926 03:11:23.422539 21553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:11:23.519029 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 21491
W20260926 03:11:23.533771 21476 connection.cc:535] client connection to 127.0.0.1:45611 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45611): Connection reset by peer (error 104)
2026-09-26T03:11:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1971 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:11:23 dist-test-slave-hmbw krb5kdc[21591](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:11:23 dist-test-slave-hmbw krb5kdc[21591](info): set up 2 sockets
Sep 26 03:11:23 dist-test-slave-hmbw krb5kdc[21591](info): commencing operation
krb5kdc: starting...
W20260926 03:11:24.538718 388 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:11:24 dist-test-slave-hmbw krb5kdc[21591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:11:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:11:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:11:24.791208 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35557
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:11:25.123703 21604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:25.124044 21604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:25.124184 21604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:25.131574 21604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:11:25.131727 21604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:25.131803 21604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:25.131893 21604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:11:25.131963 21604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:11:25.142072 21604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:25.143501 21604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:25.144822 21604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:25.156450 21609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:25.156405 21612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:25.156498 21610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:25.157214 21604 server_base.cc:1034] running on GCE node
I20260926 03:11:25.157979 21604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:25.159482 21604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:25.160699 21604 hybrid_clock.cc:648] HybridClock initialized: now 1790392285160641 us; error 52 us; skew 500 ppm
I20260926 03:11:25.161090 21604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:11:25 dist-test-slave-hmbw krb5kdc[21591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:11:25.164268 21604 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:11:25.165828 21604 webserver.cc:460] Webserver started at http://127.0.0.1:35039/ using document root <none> and password file <none>
I20260926 03:11:25.166570 21604 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:25.166684 21604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:25.166972 21604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:25.169937 21604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "bdf59e4d9b51406eb1425675ec588c27"
format_stamp: "Formatted at 2026-09-26 03:11:25 on dist-test-slave-hmbw"
server_key: "0f3e3975fa11dbd77378daa032596842"
server_key_iv: "a6811a996bd1168463212f0dc3c8c4aa"
server_key_version: "encryptionkey@0"
I20260926 03:11:25.170764 21604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "bdf59e4d9b51406eb1425675ec588c27"
format_stamp: "Formatted at 2026-09-26 03:11:25 on dist-test-slave-hmbw"
server_key: "0f3e3975fa11dbd77378daa032596842"
server_key_iv: "a6811a996bd1168463212f0dc3c8c4aa"
server_key_version: "encryptionkey@0"
I20260926 03:11:25.176222 21604 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:11:25.179943 21619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:25.181380 21604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:11:25.181548 21604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "bdf59e4d9b51406eb1425675ec588c27"
format_stamp: "Formatted at 2026-09-26 03:11:25 on dist-test-slave-hmbw"
server_key: "0f3e3975fa11dbd77378daa032596842"
server_key_iv: "a6811a996bd1168463212f0dc3c8c4aa"
server_key_version: "encryptionkey@0"
I20260926 03:11:25.181762 21604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:25.196873 21604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:25.201207 21604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:25.201473 21604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:25.219551 21604 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35557
I20260926 03:11:25.219580 21670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35557 every 8 connection(s)
I20260926 03:11:25.220989 21604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:11:25.224750 21671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:25.225570 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 21604
I20260926 03:11:25.225716 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:11:25.226130 388 external_mini_cluster.cc:1428] Setting key 2514135fd03bf1fd5952f08a18734268
I20260926 03:11:25.237486 21671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdf59e4d9b51406eb1425675ec588c27: Bootstrap starting.
I20260926 03:11:25.239923 21674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:25.227950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:47201 (local address 127.0.0.1:35557)
0926 03:11:25.228732 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:25.228744 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:25.228784 (+ 40us) server_negotiation.cc:407] Connection header received
0926 03:11:25.230919 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:25.230945 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:25.231465 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:25.231876 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:25.232442 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:25.233692 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:25.235185 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:25.235928 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:25.236096 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:25.237526 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:25.237535 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:25.237551 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:25.238143 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:25.238242 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:25.238271 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:25.238420 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:25.238754 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 03:11:25.238942 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":341,"threads_started":1}
I20260926 03:11:25.241004 21671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdf59e4d9b51406eb1425675ec588c27: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:25.242079 21671 log.cc:824] T 00000000000000000000000000000000 P bdf59e4d9b51406eb1425675ec588c27: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:25.243925 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:11:25.244980 21671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdf59e4d9b51406eb1425675ec588c27: No bootstrap required, opened a new log
I20260926 03:11:25.253296 21671 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdf59e4d9b51406eb1425675ec588c27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf59e4d9b51406eb1425675ec588c27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35557 } }
I20260926 03:11:25.253759 21671 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdf59e4d9b51406eb1425675ec588c27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:25.253876 21671 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdf59e4d9b51406eb1425675ec588c27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf59e4d9b51406eb1425675ec588c27, State: Initialized, Role: FOLLOWER
I20260926 03:11:25.254323 21674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:25.248143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:33401 (local address 127.0.0.1:35557)
0926 03:11:25.248329 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:25.248336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:25.248649 (+ 313us) server_negotiation.cc:407] Connection header received
0926 03:11:25.248838 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:25.248842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:25.248913 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:25.249002 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:25.249528 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:25.250315 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:25.251740 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:25.252249 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:25.252376 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:25.253561 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:25.253566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:25.253570 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:25.253590 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:25.253633 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:25.253649 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:25.253742 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:25.253944 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 03:11:25.253997 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 03:11:25.254676 21671 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdf59e4d9b51406eb1425675ec588c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf59e4d9b51406eb1425675ec588c27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35557 } }
I20260926 03:11:25.254935 21671 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdf59e4d9b51406eb1425675ec588c27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:11:25.255066 21671 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdf59e4d9b51406eb1425675ec588c27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:11:25.255232 21671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdf59e4d9b51406eb1425675ec588c27 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:25.257709 21671 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdf59e4d9b51406eb1425675ec588c27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf59e4d9b51406eb1425675ec588c27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35557 } }
I20260926 03:11:25.258265 21671 leader_election.cc:302] T 00000000000000000000000000000000 P bdf59e4d9b51406eb1425675ec588c27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdf59e4d9b51406eb1425675ec588c27; no voters:
I20260926 03:11:25.259117 21671 leader_election.cc:288] T 00000000000000000000000000000000 P bdf59e4d9b51406eb1425675ec588c27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:11:25.259295 21681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdf59e4d9b51406eb1425675ec588c27 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:11:25.259963 21681 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdf59e4d9b51406eb1425675ec588c27 [term 1 LEADER]: Becoming Leader. State: Replica: bdf59e4d9b51406eb1425675ec588c27, State: Running, Role: LEADER
I20260926 03:11:25.260571 21681 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdf59e4d9b51406eb1425675ec588c27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf59e4d9b51406eb1425675ec588c27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35557 } }
I20260926 03:11:25.261765 21671 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdf59e4d9b51406eb1425675ec588c27 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:11:25.263425 21683 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdf59e4d9b51406eb1425675ec588c27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdf59e4d9b51406eb1425675ec588c27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf59e4d9b51406eb1425675ec588c27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35557 } } }
I20260926 03:11:25.263860 21683 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdf59e4d9b51406eb1425675ec588c27 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:25.264395 21682 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdf59e4d9b51406eb1425675ec588c27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdf59e4d9b51406eb1425675ec588c27. Latest consensus state: current_term: 1 leader_uuid: "bdf59e4d9b51406eb1425675ec588c27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf59e4d9b51406eb1425675ec588c27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35557 } } }
I20260926 03:11:25.264657 21686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:11:25.264706 21682 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdf59e4d9b51406eb1425675ec588c27 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:25.270002 21686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:11:25.281514 21686 catalog_manager.cc:1347] Generated new cluster ID: bb41cccdba144c8b986bea1f0e903e21
I20260926 03:11:25.281649 21686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:11:25.306968 21686 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:11:25.308889 21686 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:11:25.319842 21686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdf59e4d9b51406eb1425675ec588c27: Generated new TSK 0
I20260926 03:11:25.321009 21686 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:11:25.334954 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 21604
W20260926 03:11:25.336222 21666 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.111:47201
W20260926 03:11:25.337038 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 21604: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:11:25.337199 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 21604
W20260926 03:11:25.351514 21589 connection.cc:535] client connection to 127.0.0.1:35557 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35557): Connection reset by peer (error 104)
2026-09-26T03:11:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1817 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:11:25 dist-test-slave-hmbw krb5kdc[21702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:11:25 dist-test-slave-hmbw krb5kdc[21702](info): set up 2 sockets
Sep 26 03:11:25 dist-test-slave-hmbw krb5kdc[21702](info): commencing operation
krb5kdc: starting...
W20260926 03:11:26.358736 388 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:11:26 dist-test-slave-hmbw krb5kdc[21702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:11:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:11:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:11:26.610363 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33467
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:11:26.940469 21715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:26.940773 21715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:26.940893 21715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:26.948372 21715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:11:26.948510 21715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:26.948575 21715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:26.948660 21715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:11:26.948719 21715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:11:26.958896 21715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:26.960220 21715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:26.961536 21715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:26.973764 21720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:26.973881 21721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:26.973793 21723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:26.975250 21715 server_base.cc:1034] running on GCE node
I20260926 03:11:26.976143 21715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:26.977665 21715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:26.978906 21715 hybrid_clock.cc:648] HybridClock initialized: now 1790392286978852 us; error 54 us; skew 500 ppm
I20260926 03:11:26.979302 21715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:11:26 dist-test-slave-hmbw krb5kdc[21702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:11:26.988058 21715 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:11:26.989815 21715 webserver.cc:460] Webserver started at http://127.0.0.1:43313/ using document root <none> and password file <none>
I20260926 03:11:26.990576 21715 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:26.990711 21715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:26.991024 21715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:26.994256 21715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "a7c8a5151e51451fa298aeb5fe699215"
format_stamp: "Formatted at 2026-09-26 03:11:26 on dist-test-slave-hmbw"
server_key: "c0d49e61e333e03cc15f17d9b275d1b7"
server_key_iv: "2647b2130b3b8e6a9af10e4d5c2449e8"
server_key_version: "encryptionkey@0"
I20260926 03:11:26.995110 21715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "a7c8a5151e51451fa298aeb5fe699215"
format_stamp: "Formatted at 2026-09-26 03:11:26 on dist-test-slave-hmbw"
server_key: "c0d49e61e333e03cc15f17d9b275d1b7"
server_key_iv: "2647b2130b3b8e6a9af10e4d5c2449e8"
server_key_version: "encryptionkey@0"
I20260926 03:11:27.000641 21715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:11:27.004629 21730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:27.006225 21715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:11:27.006525 21715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "a7c8a5151e51451fa298aeb5fe699215"
format_stamp: "Formatted at 2026-09-26 03:11:26 on dist-test-slave-hmbw"
server_key: "c0d49e61e333e03cc15f17d9b275d1b7"
server_key_iv: "2647b2130b3b8e6a9af10e4d5c2449e8"
server_key_version: "encryptionkey@0"
I20260926 03:11:27.006750 21715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:27.012928 21715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:27.017535 21715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:27.017848 21715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:27.037784 21715 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33467
I20260926 03:11:27.037811 21781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33467 every 8 connection(s)
I20260926 03:11:27.039381 21715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:11:27.043495 21782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:27.045553 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 21715
I20260926 03:11:27.045754 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:11:27.046257 388 external_mini_cluster.cc:1428] Setting key eafeb44bc919ca16eb753df3985ffb9d
I20260926 03:11:27.054801 21785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:27.048350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:42491 (local address 127.0.0.1:33467)
0926 03:11:27.049157 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:27.049170 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:27.049229 (+ 59us) server_negotiation.cc:407] Connection header received
0926 03:11:27.051027 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:27.051053 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:27.051440 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:27.051790 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:27.052944 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:27.052951 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:27.052965 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:27.052982 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:27.053053 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:27.053076 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:27.053269 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:27.053423 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 03:11:27.053671 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":359,"threads_started":1}
I20260926 03:11:27.056336 21782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7c8a5151e51451fa298aeb5fe699215: Bootstrap starting.
I20260926 03:11:27.059540 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:11:27.060992 21782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7c8a5151e51451fa298aeb5fe699215: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:27.062402 21782 log.cc:824] T 00000000000000000000000000000000 P a7c8a5151e51451fa298aeb5fe699215: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:27.066892 21782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7c8a5151e51451fa298aeb5fe699215: No bootstrap required, opened a new log
I20260926 03:11:27.067749 21785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:27.065054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:60257 (local address 127.0.0.1:33467)
0926 03:11:27.065419 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:27.065441 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:27.065467 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:11:27.065695 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:27.065701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:27.065770 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:27.065906 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:27.066953 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:27.066959 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:27.066964 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:27.066984 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:27.067046 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:27.067069 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:27.067187 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:27.067377 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 03:11:27.067449 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 03:11:27.075057 21782 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7c8a5151e51451fa298aeb5fe699215 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c8a5151e51451fa298aeb5fe699215" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33467 } }
I20260926 03:11:27.075501 21782 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7c8a5151e51451fa298aeb5fe699215 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:27.075605 21782 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7c8a5151e51451fa298aeb5fe699215 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7c8a5151e51451fa298aeb5fe699215, State: Initialized, Role: FOLLOWER
I20260926 03:11:27.076216 21782 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7c8a5151e51451fa298aeb5fe699215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c8a5151e51451fa298aeb5fe699215" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33467 } }
I20260926 03:11:27.076458 21782 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7c8a5151e51451fa298aeb5fe699215 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:11:27.076598 21782 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7c8a5151e51451fa298aeb5fe699215 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:11:27.076749 21782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7c8a5151e51451fa298aeb5fe699215 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:27.079366 21782 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7c8a5151e51451fa298aeb5fe699215 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c8a5151e51451fa298aeb5fe699215" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33467 } }
I20260926 03:11:27.079959 21782 leader_election.cc:302] T 00000000000000000000000000000000 P a7c8a5151e51451fa298aeb5fe699215 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7c8a5151e51451fa298aeb5fe699215; no voters:
I20260926 03:11:27.080796 21782 leader_election.cc:288] T 00000000000000000000000000000000 P a7c8a5151e51451fa298aeb5fe699215 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:11:27.080942 21792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7c8a5151e51451fa298aeb5fe699215 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:11:27.081684 21792 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7c8a5151e51451fa298aeb5fe699215 [term 1 LEADER]: Becoming Leader. State: Replica: a7c8a5151e51451fa298aeb5fe699215, State: Running, Role: LEADER
I20260926 03:11:27.082240 21792 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7c8a5151e51451fa298aeb5fe699215 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c8a5151e51451fa298aeb5fe699215" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33467 } }
I20260926 03:11:27.082912 21782 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7c8a5151e51451fa298aeb5fe699215 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:11:27.085479 21793 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7c8a5151e51451fa298aeb5fe699215 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7c8a5151e51451fa298aeb5fe699215" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c8a5151e51451fa298aeb5fe699215" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33467 } } }
I20260926 03:11:27.085775 21794 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7c8a5151e51451fa298aeb5fe699215 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7c8a5151e51451fa298aeb5fe699215. Latest consensus state: current_term: 1 leader_uuid: "a7c8a5151e51451fa298aeb5fe699215" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c8a5151e51451fa298aeb5fe699215" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33467 } } }
I20260926 03:11:27.086206 21794 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7c8a5151e51451fa298aeb5fe699215 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:27.087033 21793 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7c8a5151e51451fa298aeb5fe699215 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:27.087357 21799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:11:27.093160 21799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:11:27.107726 21799 catalog_manager.cc:1347] Generated new cluster ID: dceb8b8c8e644b9ca99d59c661519c62
I20260926 03:11:27.107872 21799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:11:27.135716 21799 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:11:27.136420 21799 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:11:27.144536 21799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7c8a5151e51451fa298aeb5fe699215: Generated new TSK 0
I20260926 03:11:27.145485 21799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:11:27.157943 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 21715
W20260926 03:11:27.159463 21777 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.111:42491
W20260926 03:11:27.160233 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 21715: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:11:27.160399 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 21715
2026-09-26T03:11:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1822 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:11:27 dist-test-slave-hmbw krb5kdc[21813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:11:27 dist-test-slave-hmbw krb5kdc[21813](info): set up 2 sockets
Sep 26 03:11:27 dist-test-slave-hmbw krb5kdc[21813](info): commencing operation
krb5kdc: starting...
W20260926 03:11:28.182740 388 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:11:28 dist-test-slave-hmbw krb5kdc[21813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:11:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:11:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:11:28.435232 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32769
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:11:28.765342 21826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:28.765690 21826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:28.765826 21826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:28.773303 21826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:11:28.773444 21826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:28.773512 21826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:28.773593 21826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:11:28.773648 21826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:11:28.783968 21826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32769
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:28.785318 21826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:28.786625 21826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:28.798271 21831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:28.798316 21832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:28.798475 21834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:28.798924 21826 server_base.cc:1034] running on GCE node
I20260926 03:11:28.799736 21826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:28.801169 21826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:28.802416 21826 hybrid_clock.cc:648] HybridClock initialized: now 1790392288802351 us; error 56 us; skew 500 ppm
I20260926 03:11:28.802806 21826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:11:28 dist-test-slave-hmbw krb5kdc[21813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:11:28.806074 21826 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:11:28.807665 21826 webserver.cc:460] Webserver started at http://127.0.0.1:35121/ using document root <none> and password file <none>
I20260926 03:11:28.808391 21826 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:28.808511 21826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:28.808851 21826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:28.811954 21826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "da89d949c0e04a928124359dac0399dd"
format_stamp: "Formatted at 2026-09-26 03:11:28 on dist-test-slave-hmbw"
server_key: "6e617a5c1f0a06b15e6ffc44c655a1b0"
server_key_iv: "91f97332682b9c30a85fdcb024c2d021"
server_key_version: "encryptionkey@0"
I20260926 03:11:28.812733 21826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "da89d949c0e04a928124359dac0399dd"
format_stamp: "Formatted at 2026-09-26 03:11:28 on dist-test-slave-hmbw"
server_key: "6e617a5c1f0a06b15e6ffc44c655a1b0"
server_key_iv: "91f97332682b9c30a85fdcb024c2d021"
server_key_version: "encryptionkey@0"
I20260926 03:11:28.817965 21826 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:11:28.821642 21841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:28.823334 21826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:11:28.823513 21826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "da89d949c0e04a928124359dac0399dd"
format_stamp: "Formatted at 2026-09-26 03:11:28 on dist-test-slave-hmbw"
server_key: "6e617a5c1f0a06b15e6ffc44c655a1b0"
server_key_iv: "91f97332682b9c30a85fdcb024c2d021"
server_key_version: "encryptionkey@0"
I20260926 03:11:28.823700 21826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:28.841018 21826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:28.845523 21826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:28.845801 21826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:28.863694 21826 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32769
I20260926 03:11:28.863719 21892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32769 every 8 connection(s)
I20260926 03:11:28.865057 21826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:11:28.868757 21893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:28.869740 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 21826
I20260926 03:11:28.869904 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:11:28.870299 388 external_mini_cluster.cc:1428] Setting key 444b507635202c9b7445d66eec7f8b9a
I20260926 03:11:28.881206 21893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da89d949c0e04a928124359dac0399dd: Bootstrap starting.
Sep 26 03:11:28 dist-test-slave-hmbw krb5kdc[21813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 03:11:28.884351 21893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da89d949c0e04a928124359dac0399dd: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:28.885401 21893 log.cc:824] T 00000000000000000000000000000000 P da89d949c0e04a928124359dac0399dd: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:28.888554 21893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da89d949c0e04a928124359dac0399dd: No bootstrap required, opened a new log
I20260926 03:11:28.892547 21896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:28.872027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:39715 (local address 127.0.0.1:32769)
0926 03:11:28.872806 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:28.872819 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:28.872890 (+ 71us) server_negotiation.cc:407] Connection header received
0926 03:11:28.874932 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:28.874961 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:28.875409 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:28.875817 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:28.877218 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:28.878598 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:28.879871 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:28.880563 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:28.880772 (+ 209us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:28.884890 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:28.884900 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:28.884916 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:28.884959 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:28.888292 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:28.889155 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:28.889162 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:28.889168 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:28.889342 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:28.889805 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:28.889809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:28.889811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:28.890382 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:28.890724 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:28.891211 (+ 487us) server_negotiation.cc:299] Negotiation successful
0926 03:11:28.891526 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":316,"threads_started":1}
I20260926 03:11:28.896469 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:11:28.896610 21893 raft_consensus.cc:348] T 00000000000000000000000000000000 P da89d949c0e04a928124359dac0399dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da89d949c0e04a928124359dac0399dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32769 } }
I20260926 03:11:28.896991 21893 raft_consensus.cc:374] T 00000000000000000000000000000000 P da89d949c0e04a928124359dac0399dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:28.897089 21893 raft_consensus.cc:729] T 00000000000000000000000000000000 P da89d949c0e04a928124359dac0399dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da89d949c0e04a928124359dac0399dd, State: Initialized, Role: FOLLOWER
I20260926 03:11:28.897644 21893 consensus_queue.cc:260] T 00000000000000000000000000000000 P da89d949c0e04a928124359dac0399dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da89d949c0e04a928124359dac0399dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32769 } }
I20260926 03:11:28.897871 21893 raft_consensus.cc:388] T 00000000000000000000000000000000 P da89d949c0e04a928124359dac0399dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:11:28.897993 21893 raft_consensus.cc:482] T 00000000000000000000000000000000 P da89d949c0e04a928124359dac0399dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:11:28.898134 21893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da89d949c0e04a928124359dac0399dd [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:28.900652 21893 raft_consensus.cc:504] T 00000000000000000000000000000000 P da89d949c0e04a928124359dac0399dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da89d949c0e04a928124359dac0399dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32769 } }
I20260926 03:11:28.901262 21893 leader_election.cc:302] T 00000000000000000000000000000000 P da89d949c0e04a928124359dac0399dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da89d949c0e04a928124359dac0399dd; no voters:
I20260926 03:11:28.901870 21893 leader_election.cc:288] T 00000000000000000000000000000000 P da89d949c0e04a928124359dac0399dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:11:28.901973 21903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da89d949c0e04a928124359dac0399dd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:11:28.903970 21893 sys_catalog.cc:564] T 00000000000000000000000000000000 P da89d949c0e04a928124359dac0399dd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:11:28.906791 21903 raft_consensus.cc:686] T 00000000000000000000000000000000 P da89d949c0e04a928124359dac0399dd [term 1 LEADER]: Becoming Leader. State: Replica: da89d949c0e04a928124359dac0399dd, State: Running, Role: LEADER
I20260926 03:11:28.907367 21903 consensus_queue.cc:237] T 00000000000000000000000000000000 P da89d949c0e04a928124359dac0399dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da89d949c0e04a928124359dac0399dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32769 } }
I20260926 03:11:28.909960 21904 sys_catalog.cc:455] T 00000000000000000000000000000000 P da89d949c0e04a928124359dac0399dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da89d949c0e04a928124359dac0399dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da89d949c0e04a928124359dac0399dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32769 } } }
I20260926 03:11:28.909902 21905 sys_catalog.cc:455] T 00000000000000000000000000000000 P da89d949c0e04a928124359dac0399dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader da89d949c0e04a928124359dac0399dd. Latest consensus state: current_term: 1 leader_uuid: "da89d949c0e04a928124359dac0399dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da89d949c0e04a928124359dac0399dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32769 } } }
I20260926 03:11:28.910422 21904 sys_catalog.cc:458] T 00000000000000000000000000000000 P da89d949c0e04a928124359dac0399dd [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:28.910422 21905 sys_catalog.cc:458] T 00000000000000000000000000000000 P da89d949c0e04a928124359dac0399dd [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:28.919044 21909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:11:28.920974 21896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:28.900996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:55517 (local address 127.0.0.1:32769)
0926 03:11:28.901194 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:28.901200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:28.901558 (+ 358us) server_negotiation.cc:407] Connection header received
0926 03:11:28.901757 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:28.901762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:28.901858 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:28.901973 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:28.904028 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:28.905164 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:28.907453 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:28.908174 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:28.908290 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:28.911982 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:28.911989 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:28.911997 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:28.912051 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:28.916659 (+ 4608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:28.918492 (+ 1833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:28.918519 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:28.918522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:28.918700 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:28.919300 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:28.919305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:28.919309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:28.919735 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:28.920033 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:28.920417 (+ 384us) server_negotiation.cc:299] Negotiation successful
0926 03:11:28.920618 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:11:28.930368 21909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:11:28.939328 21909 catalog_manager.cc:1347] Generated new cluster ID: e49c8170f9914be78a84a94386b7f475
I20260926 03:11:28.939455 21909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:11:28.964699 21909 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:11:28.966600 21909 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:11:28.976112 21909 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da89d949c0e04a928124359dac0399dd: Generated new TSK 0
I20260926 03:11:28.977280 21909 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:11:28.984079 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 21826
I20260926 03:11:28.985968 21888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:11:29.081805 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 21826
W20260926 03:11:29.095971 21811 connection.cc:535] client connection to 127.0.0.1:32769 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32769): Connection reset by peer (error 104)
2026-09-26T03:11:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1922 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:11:29 dist-test-slave-hmbw krb5kdc[21925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:11:29 dist-test-slave-hmbw krb5kdc[21925](info): set up 2 sockets
Sep 26 03:11:29 dist-test-slave-hmbw krb5kdc[21925](info): commencing operation
krb5kdc: starting...
W20260926 03:11:30.111030 388 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:11:30 dist-test-slave-hmbw krb5kdc[21925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:11:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:11:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:11:30.375115 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40093
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:11:30.728305 21938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:30.728638 21938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:30.728793 21938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:30.740361 21938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:11:30.740531 21938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:30.740657 21938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:30.740751 21938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:11:30.740814 21938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:11:30.753616 21938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40093
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:30.755139 21938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:30.756578 21938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:30.769585 21944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:30.769555 21943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:30.769646 21946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:30.770339 21938 server_base.cc:1034] running on GCE node
I20260926 03:11:30.771224 21938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:30.772789 21938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:30.774034 21938 hybrid_clock.cc:648] HybridClock initialized: now 1790392290773994 us; error 34 us; skew 500 ppm
I20260926 03:11:30.774422 21938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:11:30 dist-test-slave-hmbw krb5kdc[21925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:11:30.777803 21938 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:11:30.779534 21938 webserver.cc:460] Webserver started at http://127.0.0.1:43379/ using document root <none> and password file <none>
I20260926 03:11:30.780267 21938 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:30.780377 21938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:30.780694 21938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:30.783936 21938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "d907ee8904d046dba10b962547fc1eca"
format_stamp: "Formatted at 2026-09-26 03:11:30 on dist-test-slave-hmbw"
server_key: "092ce7edc77d864cc8603c4d022487bb"
server_key_iv: "4a47e94e7c53aa64458e47ec21542ae7"
server_key_version: "encryptionkey@0"
I20260926 03:11:30.784744 21938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "d907ee8904d046dba10b962547fc1eca"
format_stamp: "Formatted at 2026-09-26 03:11:30 on dist-test-slave-hmbw"
server_key: "092ce7edc77d864cc8603c4d022487bb"
server_key_iv: "4a47e94e7c53aa64458e47ec21542ae7"
server_key_version: "encryptionkey@0"
I20260926 03:11:30.790699 21938 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 03:11:30.794848 21953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:30.796368 21938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:11:30.796528 21938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "d907ee8904d046dba10b962547fc1eca"
format_stamp: "Formatted at 2026-09-26 03:11:30 on dist-test-slave-hmbw"
server_key: "092ce7edc77d864cc8603c4d022487bb"
server_key_iv: "4a47e94e7c53aa64458e47ec21542ae7"
server_key_version: "encryptionkey@0"
I20260926 03:11:30.796711 21938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:30.818404 21938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:30.823212 21938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:30.823482 21938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:30.842049 21938 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40093
I20260926 03:11:30.842097 22004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40093 every 8 connection(s)
I20260926 03:11:30.843520 21938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:11:30.847527 22005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:30.852349 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 21938
I20260926 03:11:30.852517 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:11:30.852960 388 external_mini_cluster.cc:1428] Setting key 2306cdc7ed57ac66e24a1667280ead91
I20260926 03:11:30.860046 22005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d907ee8904d046dba10b962547fc1eca: Bootstrap starting.
I20260926 03:11:30.863997 22005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d907ee8904d046dba10b962547fc1eca: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:30.865182 22005 log.cc:824] T 00000000000000000000000000000000 P d907ee8904d046dba10b962547fc1eca: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:30.868891 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:30.855288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:55945 (local address 127.0.0.1:40093)
0926 03:11:30.856250 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:30.856266 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:30.856314 (+ 48us) server_negotiation.cc:407] Connection header received
0926 03:11:30.858403 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:30.858439 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:30.859036 (+ 597us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:30.859475 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:30.860119 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:30.861925 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:30.863493 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:30.864335 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:30.864551 (+ 216us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:30.865903 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:30.865913 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:30.865930 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:30.866684 (+ 754us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:30.866803 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:30.866855 (+ 52us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:30.867070 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:30.867599 (+ 529us) server_negotiation.cc:299] Negotiation successful
0926 03:11:30.867802 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":420,"threads_started":1}
I20260926 03:11:30.869443 22005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d907ee8904d046dba10b962547fc1eca: No bootstrap required, opened a new log
I20260926 03:11:30.873396 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:11:30.879781 22005 raft_consensus.cc:348] T 00000000000000000000000000000000 P d907ee8904d046dba10b962547fc1eca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d907ee8904d046dba10b962547fc1eca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40093 } }
I20260926 03:11:30.880362 22005 raft_consensus.cc:374] T 00000000000000000000000000000000 P d907ee8904d046dba10b962547fc1eca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:30.880479 22005 raft_consensus.cc:729] T 00000000000000000000000000000000 P d907ee8904d046dba10b962547fc1eca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d907ee8904d046dba10b962547fc1eca, State: Initialized, Role: FOLLOWER
I20260926 03:11:30.881274 22005 consensus_queue.cc:260] T 00000000000000000000000000000000 P d907ee8904d046dba10b962547fc1eca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d907ee8904d046dba10b962547fc1eca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40093 } }
I20260926 03:11:30.881584 22005 raft_consensus.cc:388] T 00000000000000000000000000000000 P d907ee8904d046dba10b962547fc1eca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:11:30.881747 22005 raft_consensus.cc:482] T 00000000000000000000000000000000 P d907ee8904d046dba10b962547fc1eca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:11:30.881922 22005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d907ee8904d046dba10b962547fc1eca [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:30.884716 22005 raft_consensus.cc:504] T 00000000000000000000000000000000 P d907ee8904d046dba10b962547fc1eca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d907ee8904d046dba10b962547fc1eca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40093 } }
I20260926 03:11:30.885354 22005 leader_election.cc:302] T 00000000000000000000000000000000 P d907ee8904d046dba10b962547fc1eca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d907ee8904d046dba10b962547fc1eca; no voters:
I20260926 03:11:30.886138 22005 leader_election.cc:288] T 00000000000000000000000000000000 P d907ee8904d046dba10b962547fc1eca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:11:30.886147 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:30.879359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:49713 (local address 127.0.0.1:40093)
0926 03:11:30.879558 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:30.879564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:30.879872 (+ 308us) server_negotiation.cc:407] Connection header received
0926 03:11:30.880033 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:30.880038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:30.880129 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:30.880270 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:30.880784 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:30.881933 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:30.883315 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:30.883977 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:30.884104 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:30.885229 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:30.885236 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:30.885241 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:30.885266 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:30.885342 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:30.885365 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:30.885490 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:30.885715 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 03:11:30.885776 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 03:11:30.886291 22015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d907ee8904d046dba10b962547fc1eca [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:11:30.887009 22015 raft_consensus.cc:686] T 00000000000000000000000000000000 P d907ee8904d046dba10b962547fc1eca [term 1 LEADER]: Becoming Leader. State: Replica: d907ee8904d046dba10b962547fc1eca, State: Running, Role: LEADER
I20260926 03:11:30.887735 22015 consensus_queue.cc:237] T 00000000000000000000000000000000 P d907ee8904d046dba10b962547fc1eca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d907ee8904d046dba10b962547fc1eca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40093 } }
I20260926 03:11:30.889587 22005 sys_catalog.cc:564] T 00000000000000000000000000000000 P d907ee8904d046dba10b962547fc1eca [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:11:30.890597 22017 sys_catalog.cc:455] T 00000000000000000000000000000000 P d907ee8904d046dba10b962547fc1eca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d907ee8904d046dba10b962547fc1eca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d907ee8904d046dba10b962547fc1eca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40093 } } }
I20260926 03:11:30.891016 22017 sys_catalog.cc:458] T 00000000000000000000000000000000 P d907ee8904d046dba10b962547fc1eca [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:30.892372 22016 sys_catalog.cc:455] T 00000000000000000000000000000000 P d907ee8904d046dba10b962547fc1eca [sys.catalog]: SysCatalogTable state changed. Reason: New leader d907ee8904d046dba10b962547fc1eca. Latest consensus state: current_term: 1 leader_uuid: "d907ee8904d046dba10b962547fc1eca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d907ee8904d046dba10b962547fc1eca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40093 } } }
I20260926 03:11:30.892647 22020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:11:30.892743 22016 sys_catalog.cc:458] T 00000000000000000000000000000000 P d907ee8904d046dba10b962547fc1eca [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:30.897974 22020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:11:30.908622 22020 catalog_manager.cc:1347] Generated new cluster ID: 8b4c0333c3e74b06b37a2fdbee694cdd
I20260926 03:11:30.908797 22020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:11:30.940445 22020 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:11:30.942096 22020 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:11:30.949985 22020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d907ee8904d046dba10b962547fc1eca: Generated new TSK 0
I20260926 03:11:30.950878 22020 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:11:30.959322 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 21938
W20260926 03:11:30.960713 22000 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.111:55945
W20260926 03:11:30.961527 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 21938: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:11:30.961691 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 21938
W20260926 03:11:30.975896 21923 connection.cc:535] client connection to 127.0.0.1:40093 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40093): Connection reset by peer (error 104)
2026-09-26T03:11:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1883 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:11:31 dist-test-slave-hmbw krb5kdc[22036](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:11:31 dist-test-slave-hmbw krb5kdc[22036](info): set up 2 sockets
Sep 26 03:11:31 dist-test-slave-hmbw krb5kdc[22036](info): commencing operation
krb5kdc: starting...
W20260926 03:11:31.994952 388 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.011s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:11:32 dist-test-slave-hmbw krb5kdc[22036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:11:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:11:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:11:32.254575 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36083
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:11:32.586028 22049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:32.586371 22049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:32.586526 22049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:32.593998 22049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:11:32.594139 22049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:32.594236 22049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:32.594322 22049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:11:32.594383 22049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:11:32.604719 22049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36083
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391950058021-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:32.606052 22049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:32.607426 22049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:32.619976 22057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:32.620042 22055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:32.620012 22054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:32.621423 22049 server_base.cc:1034] running on GCE node
I20260926 03:11:32.622444 22049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:32.624464 22049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:32.627806 22049 hybrid_clock.cc:648] HybridClock initialized: now 1790392292627726 us; error 68 us; skew 500 ppm
I20260926 03:11:32.628242 22049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:11:32 dist-test-slave-hmbw krb5kdc[22036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:11:32.637601 22049 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:11:32.639437 22049 webserver.cc:460] Webserver started at http://127.0.0.1:33103/ using document root <none> and password file <none>
I20260926 03:11:32.640177 22049 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:32.640296 22049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:32.640571 22049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:32.643699 22049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "cbf9ea393b3047aca3b5985a4c73f5c9"
format_stamp: "Formatted at 2026-09-26 03:11:32 on dist-test-slave-hmbw"
server_key: "584c3b7c4f1f3faf1b564f7274fa71d6"
server_key_iv: "296b50b75af41228353713feaf46034e"
server_key_version: "encryptionkey@0"
I20260926 03:11:32.644469 22049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "cbf9ea393b3047aca3b5985a4c73f5c9"
format_stamp: "Formatted at 2026-09-26 03:11:32 on dist-test-slave-hmbw"
server_key: "584c3b7c4f1f3faf1b564f7274fa71d6"
server_key_iv: "296b50b75af41228353713feaf46034e"
server_key_version: "encryptionkey@0"
I20260926 03:11:32.650003 22049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:11:32.653842 22064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:32.655403 22049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:11:32.655562 22049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "cbf9ea393b3047aca3b5985a4c73f5c9"
format_stamp: "Formatted at 2026-09-26 03:11:32 on dist-test-slave-hmbw"
server_key: "584c3b7c4f1f3faf1b564f7274fa71d6"
server_key_iv: "296b50b75af41228353713feaf46034e"
server_key_version: "encryptionkey@0"
I20260926 03:11:32.655751 22049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:32.661931 22049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:32.666422 22049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:32.666810 22049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:32.685189 22049 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36083
I20260926 03:11:32.685217 22115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36083 every 8 connection(s)
I20260926 03:11:32.686787 22049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:11:32.689484 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 22049
I20260926 03:11:32.689621 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:11:32.690045 388 external_mini_cluster.cc:1428] Setting key 7266115665351585317c65585ed05bfc
I20260926 03:11:32.691090 22116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:32.699216 22119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:32.692085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:44941 (local address 127.0.0.1:36083)
0926 03:11:32.692950 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:32.692965 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:32.693015 (+ 50us) server_negotiation.cc:407] Connection header received
0926 03:11:32.694871 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:32.694904 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:32.695455 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:32.695960 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:32.697010 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:32.697018 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:32.697035 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:32.697054 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:32.697157 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:32.697186 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:32.697452 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:32.697597 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 03:11:32.697844 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":380,"threads_started":1}
I20260926 03:11:32.703238 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:11:32.706144 22116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf9ea393b3047aca3b5985a4c73f5c9: Bootstrap starting.
I20260926 03:11:32.710251 22116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbf9ea393b3047aca3b5985a4c73f5c9: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:32.711019 22119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:32.708361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:41179 (local address 127.0.0.1:36083)
0926 03:11:32.708583 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:32.708591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:32.708990 (+ 399us) server_negotiation.cc:407] Connection header received
0926 03:11:32.709140 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:32.709143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:32.709198 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:32.709341 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:32.710293 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:32.710298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:32.710301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:32.710319 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:32.710376 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:32.710400 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:32.710551 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:32.710677 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 03:11:32.710768 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:11:32.711588 22116 log.cc:824] T 00000000000000000000000000000000 P cbf9ea393b3047aca3b5985a4c73f5c9: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:32.714599 22116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf9ea393b3047aca3b5985a4c73f5c9: No bootstrap required, opened a new log
I20260926 03:11:32.722152 22116 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbf9ea393b3047aca3b5985a4c73f5c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf9ea393b3047aca3b5985a4c73f5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36083 } }
I20260926 03:11:32.722604 22116 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbf9ea393b3047aca3b5985a4c73f5c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:32.722708 22116 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbf9ea393b3047aca3b5985a4c73f5c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbf9ea393b3047aca3b5985a4c73f5c9, State: Initialized, Role: FOLLOWER
I20260926 03:11:32.723323 22116 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbf9ea393b3047aca3b5985a4c73f5c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf9ea393b3047aca3b5985a4c73f5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36083 } }
I20260926 03:11:32.723534 22116 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbf9ea393b3047aca3b5985a4c73f5c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:11:32.723656 22116 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbf9ea393b3047aca3b5985a4c73f5c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:11:32.723820 22116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbf9ea393b3047aca3b5985a4c73f5c9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:32.726080 22116 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbf9ea393b3047aca3b5985a4c73f5c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf9ea393b3047aca3b5985a4c73f5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36083 } }
I20260926 03:11:32.726706 22116 leader_election.cc:302] T 00000000000000000000000000000000 P cbf9ea393b3047aca3b5985a4c73f5c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbf9ea393b3047aca3b5985a4c73f5c9; no voters:
I20260926 03:11:32.727305 22116 leader_election.cc:288] T 00000000000000000000000000000000 P cbf9ea393b3047aca3b5985a4c73f5c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:11:32.727442 22126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbf9ea393b3047aca3b5985a4c73f5c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:11:32.728104 22126 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbf9ea393b3047aca3b5985a4c73f5c9 [term 1 LEADER]: Becoming Leader. State: Replica: cbf9ea393b3047aca3b5985a4c73f5c9, State: Running, Role: LEADER
I20260926 03:11:32.728708 22126 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbf9ea393b3047aca3b5985a4c73f5c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf9ea393b3047aca3b5985a4c73f5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36083 } }
I20260926 03:11:32.729306 22116 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbf9ea393b3047aca3b5985a4c73f5c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:11:32.731310 22128 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf9ea393b3047aca3b5985a4c73f5c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbf9ea393b3047aca3b5985a4c73f5c9. Latest consensus state: current_term: 1 leader_uuid: "cbf9ea393b3047aca3b5985a4c73f5c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf9ea393b3047aca3b5985a4c73f5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36083 } } }
I20260926 03:11:32.731954 22128 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf9ea393b3047aca3b5985a4c73f5c9 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:32.732848 22127 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf9ea393b3047aca3b5985a4c73f5c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbf9ea393b3047aca3b5985a4c73f5c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf9ea393b3047aca3b5985a4c73f5c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36083 } } }
I20260926 03:11:32.733119 22135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:11:32.733147 22127 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf9ea393b3047aca3b5985a4c73f5c9 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:32.742095 22135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:11:32.752504 22135 catalog_manager.cc:1347] Generated new cluster ID: b76b4e2131954263b27515efa63448ae
I20260926 03:11:32.752643 22135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:11:32.815516 22135 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:11:32.816068 22135 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:11:32.827262 22135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbf9ea393b3047aca3b5985a4c73f5c9: Generated new TSK 0
I20260926 03:11:32.828401 22135 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:11:32.842234 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 22049
W20260926 03:11:32.843443 22111 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.111:44941
W20260926 03:11:32.844187 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 22049: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:11:32.844343 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 22049
2026-09-26T03:11:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1878 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:11:32 dist-test-slave-hmbw krb5kdc[22147](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:11:32 dist-test-slave-hmbw krb5kdc[22147](info): set up 2 sockets
Sep 26 03:11:32 dist-test-slave-hmbw krb5kdc[22147](info): commencing operation
krb5kdc: starting...
W20260926 03:11:33.874753 388 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:11:34 dist-test-slave-hmbw krb5kdc[22147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:11:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:11:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:11:34.130069 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39155
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:11:34.463647 22160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:34.463979 22160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:34.464118 22160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:34.471911 22160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:11:34.472061 22160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:34.472143 22160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:34.472230 22160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:11:34.472289 22160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:11:34.482393 22160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39155
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:34.483770 22160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:34.485077 22160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:34.496646 22165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:34.496676 22168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:34.496611 22166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:34.497601 22160 server_base.cc:1034] running on GCE node
I20260926 03:11:34.498426 22160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:34.499984 22160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:34.501248 22160 hybrid_clock.cc:648] HybridClock initialized: now 1790392294501172 us; error 70 us; skew 500 ppm
I20260926 03:11:34.501657 22160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:11:34 dist-test-slave-hmbw krb5kdc[22147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:11:34.504878 22160 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:11:34.506474 22160 webserver.cc:460] Webserver started at http://127.0.97.62:46865/ using document root <none> and password file <none>
I20260926 03:11:34.507201 22160 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:34.507314 22160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:34.507604 22160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:34.510655 22160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "ea5a0f5e35854d4e844de7a4bcdb33b0"
format_stamp: "Formatted at 2026-09-26 03:11:34 on dist-test-slave-hmbw"
server_key: "65883b35074d963ba0f0a8e2160242d6"
server_key_iv: "919dd2464f44455e5e830310c8dbf5f4"
server_key_version: "encryptionkey@0"
I20260926 03:11:34.511471 22160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "ea5a0f5e35854d4e844de7a4bcdb33b0"
format_stamp: "Formatted at 2026-09-26 03:11:34 on dist-test-slave-hmbw"
server_key: "65883b35074d963ba0f0a8e2160242d6"
server_key_iv: "919dd2464f44455e5e830310c8dbf5f4"
server_key_version: "encryptionkey@0"
I20260926 03:11:34.516911 22160 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:11:34.520560 22175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:34.522001 22160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:11:34.522209 22160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "ea5a0f5e35854d4e844de7a4bcdb33b0"
format_stamp: "Formatted at 2026-09-26 03:11:34 on dist-test-slave-hmbw"
server_key: "65883b35074d963ba0f0a8e2160242d6"
server_key_iv: "919dd2464f44455e5e830310c8dbf5f4"
server_key_version: "encryptionkey@0"
I20260926 03:11:34.522421 22160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:34.539320 22160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:34.543445 22160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:34.543720 22160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:34.561719 22160 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39155
I20260926 03:11:34.561726 22226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39155 every 8 connection(s)
I20260926 03:11:34.563130 22160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:11:34.565514 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 22160
I20260926 03:11:34.565670 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:11:34.566116 388 external_mini_cluster.cc:1428] Setting key 4fa2111f2d67bc118ada82c83c2868fc
I20260926 03:11:34.567296 22227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 03:11:34 dist-test-slave-hmbw krb5kdc[22147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:11:34.579994 22227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea5a0f5e35854d4e844de7a4bcdb33b0: Bootstrap starting.
I20260926 03:11:34.584076 22227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea5a0f5e35854d4e844de7a4bcdb33b0: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:34.585379 22227 log.cc:824] T 00000000000000000000000000000000 P ea5a0f5e35854d4e844de7a4bcdb33b0: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:34.589104 22227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea5a0f5e35854d4e844de7a4bcdb33b0: No bootstrap required, opened a new log
I20260926 03:11:34.590109 22230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:34.567870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34167 (local address 127.0.97.62:39155)
0926 03:11:34.568706 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:34.568723 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:34.568765 (+ 42us) server_negotiation.cc:407] Connection header received
0926 03:11:34.570394 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:34.570416 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:34.570939 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:34.571389 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:34.572978 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:34.574379 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:34.575693 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:34.576385 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:34.576578 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:34.580846 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:34.580858 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:34.580885 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:34.580938 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:34.584781 (+ 3843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:34.585900 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:34.585912 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:34.585927 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:34.586188 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:34.586768 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:34.586772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:34.586774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:34.587565 (+ 791us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:34.587890 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:34.588338 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 03:11:34.588694 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":379,"threads_started":1}
I20260926 03:11:34.594148 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:11:34.597618 22227 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea5a0f5e35854d4e844de7a4bcdb33b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5a0f5e35854d4e844de7a4bcdb33b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39155 } }
I20260926 03:11:34.598101 22227 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea5a0f5e35854d4e844de7a4bcdb33b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:34.598219 22227 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea5a0f5e35854d4e844de7a4bcdb33b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea5a0f5e35854d4e844de7a4bcdb33b0, State: Initialized, Role: FOLLOWER
I20260926 03:11:34.598941 22227 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea5a0f5e35854d4e844de7a4bcdb33b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5a0f5e35854d4e844de7a4bcdb33b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39155 } }
I20260926 03:11:34.599221 22227 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea5a0f5e35854d4e844de7a4bcdb33b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:11:34.599380 22227 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea5a0f5e35854d4e844de7a4bcdb33b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:11:34.599567 22227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea5a0f5e35854d4e844de7a4bcdb33b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:34.602396 22227 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea5a0f5e35854d4e844de7a4bcdb33b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5a0f5e35854d4e844de7a4bcdb33b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39155 } }
I20260926 03:11:34.602960 22227 leader_election.cc:302] T 00000000000000000000000000000000 P ea5a0f5e35854d4e844de7a4bcdb33b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea5a0f5e35854d4e844de7a4bcdb33b0; no voters:
I20260926 03:11:34.603595 22227 leader_election.cc:288] T 00000000000000000000000000000000 P ea5a0f5e35854d4e844de7a4bcdb33b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:11:34.603699 22237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea5a0f5e35854d4e844de7a4bcdb33b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:11:34.604388 22237 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea5a0f5e35854d4e844de7a4bcdb33b0 [term 1 LEADER]: Becoming Leader. State: Replica: ea5a0f5e35854d4e844de7a4bcdb33b0, State: Running, Role: LEADER
I20260926 03:11:34.605057 22237 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea5a0f5e35854d4e844de7a4bcdb33b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5a0f5e35854d4e844de7a4bcdb33b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39155 } }
I20260926 03:11:34.606068 22227 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea5a0f5e35854d4e844de7a4bcdb33b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:11:34.607367 22238 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea5a0f5e35854d4e844de7a4bcdb33b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea5a0f5e35854d4e844de7a4bcdb33b0. Latest consensus state: current_term: 1 leader_uuid: "ea5a0f5e35854d4e844de7a4bcdb33b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5a0f5e35854d4e844de7a4bcdb33b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39155 } } }
I20260926 03:11:34.607805 22238 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea5a0f5e35854d4e844de7a4bcdb33b0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:34.607908 22239 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea5a0f5e35854d4e844de7a4bcdb33b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea5a0f5e35854d4e844de7a4bcdb33b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5a0f5e35854d4e844de7a4bcdb33b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39155 } } }
I20260926 03:11:34.608210 22239 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea5a0f5e35854d4e844de7a4bcdb33b0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:34.616364 22230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:34.598483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33135 (local address 127.0.97.62:39155)
0926 03:11:34.598765 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:34.598771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:34.599025 (+ 254us) server_negotiation.cc:407] Connection header received
0926 03:11:34.599157 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:34.599161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:34.599247 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:34.599380 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:34.600784 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:34.601970 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:34.603490 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:34.604233 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:34.605211 (+ 978us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:34.607546 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:34.607555 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:34.607561 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:34.607608 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:34.611602 (+ 3994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:34.612788 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:34.612794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:34.612798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:34.613014 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:34.613620 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:34.613625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:34.613628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:34.614032 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:34.615573 (+ 1541us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:34.615758 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 03:11:34.615964 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260926 03:11:34.620512 22242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:11:34.625867 22242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:11:34.636631 22242 catalog_manager.cc:1347] Generated new cluster ID: c5d9d6540458477fba906603976a626e
I20260926 03:11:34.636766 22242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:11:34.647014 22242 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:11:34.648341 22242 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:11:34.660477 22242 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea5a0f5e35854d4e844de7a4bcdb33b0: Generated new TSK 0
I20260926 03:11:34.661435 22242 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:11:34.667220 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 22160
I20260926 03:11:34.668394 22222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:11:34.759159 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 22160
W20260926 03:11:34.773298 22145 connection.cc:535] client connection to 127.0.97.62:39155 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:39155): Connection reset by peer (error 104)
2026-09-26T03:11:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1914 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:11:34 dist-test-slave-hmbw krb5kdc[22259](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:11:34 dist-test-slave-hmbw krb5kdc[22259](info): set up 2 sockets
Sep 26 03:11:34 dist-test-slave-hmbw krb5kdc[22259](info): commencing operation
krb5kdc: starting...
W20260926 03:11:35.778640 388 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:11:35 dist-test-slave-hmbw krb5kdc[22259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:11:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:11:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:11:36.035068 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38281
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:11:36.374254 22272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:36.374605 22272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:36.374739 22272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:36.382056 22272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:11:36.382198 22272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:36.382279 22272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:36.382359 22272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:11:36.382416 22272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:11:36.393054 22272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38281
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:36.394533 22272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:36.395937 22272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:36.408070 22277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:36.408099 22278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:36.408135 22280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:36.408878 22272 server_base.cc:1034] running on GCE node
I20260926 03:11:36.409916 22272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:36.411507 22272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:36.412751 22272 hybrid_clock.cc:648] HybridClock initialized: now 1790392296412705 us; error 45 us; skew 500 ppm
I20260926 03:11:36.413115 22272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:11:36 dist-test-slave-hmbw krb5kdc[22259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:11:36.416558 22272 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:11:36.418148 22272 webserver.cc:460] Webserver started at http://127.0.97.62:46225/ using document root <none> and password file <none>
I20260926 03:11:36.418895 22272 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:36.419014 22272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:36.419306 22272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:36.422354 22272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "7b53446882234013a7d4a585c042de39"
format_stamp: "Formatted at 2026-09-26 03:11:36 on dist-test-slave-hmbw"
server_key: "a695f84c41892ec67054ed160fe08c58"
server_key_iv: "0dc9cd3413465e1c86d6db72524ab4ee"
server_key_version: "encryptionkey@0"
I20260926 03:11:36.423157 22272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "7b53446882234013a7d4a585c042de39"
format_stamp: "Formatted at 2026-09-26 03:11:36 on dist-test-slave-hmbw"
server_key: "a695f84c41892ec67054ed160fe08c58"
server_key_iv: "0dc9cd3413465e1c86d6db72524ab4ee"
server_key_version: "encryptionkey@0"
I20260926 03:11:36.428644 22272 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 03:11:36.432538 22287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:36.434125 22272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:11:36.434345 22272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "7b53446882234013a7d4a585c042de39"
format_stamp: "Formatted at 2026-09-26 03:11:36 on dist-test-slave-hmbw"
server_key: "a695f84c41892ec67054ed160fe08c58"
server_key_iv: "0dc9cd3413465e1c86d6db72524ab4ee"
server_key_version: "encryptionkey@0"
I20260926 03:11:36.434629 22272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:36.455274 22272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:36.459846 22272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:36.460124 22272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:36.477911 22272 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38281
I20260926 03:11:36.477947 22338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38281 every 8 connection(s)
I20260926 03:11:36.479287 22272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:11:36.480261 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 22272
I20260926 03:11:36.480429 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:11:36.480818 388 external_mini_cluster.cc:1428] Setting key 8cbfd2666ba304ec5a7ec73c25caa672
I20260926 03:11:36.483681 22339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:36.497155 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:36.482613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39319 (local address 127.0.97.62:38281)
0926 03:11:36.483542 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:36.483557 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:36.483606 (+ 49us) server_negotiation.cc:407] Connection header received
0926 03:11:36.485933 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:36.485966 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:36.486563 (+ 597us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:36.487146 (+ 583us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:36.487701 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:36.489717 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:36.491361 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:36.492442 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:36.492640 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:36.493995 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:36.494006 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:36.494029 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:36.494924 (+ 895us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:36.495026 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:36.495070 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:36.495264 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:36.495586 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 03:11:36.495789 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":352,"threads_started":1}
I20260926 03:11:36.498651 22339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b53446882234013a7d4a585c042de39: Bootstrap starting.
I20260926 03:11:36.501776 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:11:36.502390 22339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b53446882234013a7d4a585c042de39: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:36.503491 22339 log.cc:824] T 00000000000000000000000000000000 P 7b53446882234013a7d4a585c042de39: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:36.507099 22339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b53446882234013a7d4a585c042de39: No bootstrap required, opened a new log
I20260926 03:11:36.513365 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:36.506301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36067 (local address 127.0.97.62:38281)
0926 03:11:36.506481 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:36.506486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:36.506863 (+ 377us) server_negotiation.cc:407] Connection header received
0926 03:11:36.507019 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:36.507024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:36.507105 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:36.507220 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:36.507794 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:36.508886 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:36.510362 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:36.511116 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:36.511260 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:36.512478 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:36.512485 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:36.512490 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:36.512515 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:36.512570 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:36.512592 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:36.512721 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:36.512950 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 03:11:36.513018 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:11:36.516474 22339 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b53446882234013a7d4a585c042de39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b53446882234013a7d4a585c042de39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38281 } }
I20260926 03:11:36.516892 22339 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b53446882234013a7d4a585c042de39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:36.516994 22339 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b53446882234013a7d4a585c042de39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b53446882234013a7d4a585c042de39, State: Initialized, Role: FOLLOWER
I20260926 03:11:36.517601 22339 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b53446882234013a7d4a585c042de39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b53446882234013a7d4a585c042de39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38281 } }
I20260926 03:11:36.517823 22339 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b53446882234013a7d4a585c042de39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:11:36.517956 22339 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b53446882234013a7d4a585c042de39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:11:36.518119 22339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b53446882234013a7d4a585c042de39 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:36.520406 22339 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b53446882234013a7d4a585c042de39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b53446882234013a7d4a585c042de39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38281 } }
I20260926 03:11:36.520948 22339 leader_election.cc:302] T 00000000000000000000000000000000 P 7b53446882234013a7d4a585c042de39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b53446882234013a7d4a585c042de39; no voters:
I20260926 03:11:36.521553 22339 leader_election.cc:288] T 00000000000000000000000000000000 P 7b53446882234013a7d4a585c042de39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:11:36.521643 22349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b53446882234013a7d4a585c042de39 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:11:36.522420 22349 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b53446882234013a7d4a585c042de39 [term 1 LEADER]: Becoming Leader. State: Replica: 7b53446882234013a7d4a585c042de39, State: Running, Role: LEADER
I20260926 03:11:36.522991 22349 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b53446882234013a7d4a585c042de39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b53446882234013a7d4a585c042de39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38281 } }
I20260926 03:11:36.523741 22339 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b53446882234013a7d4a585c042de39 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:11:36.525925 22350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b53446882234013a7d4a585c042de39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b53446882234013a7d4a585c042de39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b53446882234013a7d4a585c042de39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38281 } } }
I20260926 03:11:36.526333 22350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b53446882234013a7d4a585c042de39 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:36.526557 22351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b53446882234013a7d4a585c042de39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b53446882234013a7d4a585c042de39. Latest consensus state: current_term: 1 leader_uuid: "7b53446882234013a7d4a585c042de39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b53446882234013a7d4a585c042de39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38281 } } }
I20260926 03:11:36.526942 22351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b53446882234013a7d4a585c042de39 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:36.531100 22357 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:11:36.536459 22357 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:11:36.546718 22357 catalog_manager.cc:1347] Generated new cluster ID: c9270cbe378242f083cd3ebeddf9e2fc
I20260926 03:11:36.546861 22357 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:11:36.598764 22357 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:11:36.600131 22357 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:11:36.606544 22357 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b53446882234013a7d4a585c042de39: Generated new TSK 0
I20260926 03:11:36.607652 22357 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:11:36.627167 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 22272
W20260926 03:11:36.629041 22334 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39319
W20260926 03:11:36.629868 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 22272: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:11:36.630033 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 22272
W20260926 03:11:36.644534 22257 connection.cc:535] client connection to 127.0.97.62:38281 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:38281): Connection reset by peer (error 104)
2026-09-26T03:11:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1871 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:11:36 dist-test-slave-hmbw krb5kdc[22370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:11:36 dist-test-slave-hmbw krb5kdc[22370](info): set up 2 sockets
Sep 26 03:11:36 dist-test-slave-hmbw krb5kdc[22370](info): commencing operation
krb5kdc: starting...
W20260926 03:11:37.666833 388 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.006s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:11:37 dist-test-slave-hmbw krb5kdc[22370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:11:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:11:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:11:37.919636 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44851
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:11:38.251886 22383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:38.252179 22383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:38.252307 22383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:38.259805 22383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:11:38.259943 22383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:38.259994 22383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:38.260076 22383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:11:38.260133 22383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:11:38.270648 22383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44851
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:38.272003 22383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:38.273334 22383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:38.285202 22388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:38.285161 22389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:38.285223 22391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:38.285995 22383 server_base.cc:1034] running on GCE node
I20260926 03:11:38.286820 22383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:38.288342 22383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:38.289562 22383 hybrid_clock.cc:648] HybridClock initialized: now 1790392298289505 us; error 49 us; skew 500 ppm
I20260926 03:11:38.289911 22383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:11:38 dist-test-slave-hmbw krb5kdc[22370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:11:38.299607 22383 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:11:38.301240 22383 webserver.cc:460] Webserver started at http://127.0.97.62:44201/ using document root <none> and password file <none>
I20260926 03:11:38.301942 22383 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:38.302049 22383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:38.302309 22383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:38.305498 22383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "2aa4ed319a974e9b9e3842e4137d9f14"
format_stamp: "Formatted at 2026-09-26 03:11:38 on dist-test-slave-hmbw"
server_key: "34f63116823d112823852fc82b591942"
server_key_iv: "f3da332eaefa842832576c5db04b64b6"
server_key_version: "encryptionkey@0"
I20260926 03:11:38.306257 22383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "2aa4ed319a974e9b9e3842e4137d9f14"
format_stamp: "Formatted at 2026-09-26 03:11:38 on dist-test-slave-hmbw"
server_key: "34f63116823d112823852fc82b591942"
server_key_iv: "f3da332eaefa842832576c5db04b64b6"
server_key_version: "encryptionkey@0"
I20260926 03:11:38.311939 22383 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 03:11:38.316113 22398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:38.317919 22383 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 03:11:38.318091 22383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "2aa4ed319a974e9b9e3842e4137d9f14"
format_stamp: "Formatted at 2026-09-26 03:11:38 on dist-test-slave-hmbw"
server_key: "34f63116823d112823852fc82b591942"
server_key_iv: "f3da332eaefa842832576c5db04b64b6"
server_key_version: "encryptionkey@0"
I20260926 03:11:38.318279 22383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:38.324175 22383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:38.328536 22383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:38.328802 22383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:38.346768 22383 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44851
I20260926 03:11:38.346803 22449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44851 every 8 connection(s)
I20260926 03:11:38.348268 22383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:11:38.352169 22450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:38.354277 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 22383
I20260926 03:11:38.354454 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:11:38.355026 388 external_mini_cluster.cc:1428] Setting key 1edc1b3ca8173b0209af05e201733368
I20260926 03:11:38.363066 22453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:38.356873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44619 (local address 127.0.97.62:44851)
0926 03:11:38.357747 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:38.357763 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:38.357807 (+ 44us) server_negotiation.cc:407] Connection header received
0926 03:11:38.359495 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:38.359525 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:38.359924 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:38.360262 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:38.361304 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:38.361310 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:38.361326 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:38.361341 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:38.361394 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:38.361415 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:38.361619 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:38.361743 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 03:11:38.362027 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":392,"threads_started":1}
I20260926 03:11:38.366159 22450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aa4ed319a974e9b9e3842e4137d9f14: Bootstrap starting.
I20260926 03:11:38.367008 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:11:38.370175 22450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2aa4ed319a974e9b9e3842e4137d9f14: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:38.371598 22450 log.cc:824] T 00000000000000000000000000000000 P 2aa4ed319a974e9b9e3842e4137d9f14: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:38.374843 22453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:38.372205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43705 (local address 127.0.97.62:44851)
0926 03:11:38.372490 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:38.372496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:38.372810 (+ 314us) server_negotiation.cc:407] Connection header received
0926 03:11:38.372998 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:38.373003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:38.373061 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:38.373164 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:38.374171 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:38.374176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:38.374180 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:38.374199 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:38.374245 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:38.374266 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:38.374397 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:38.374472 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 03:11:38.374565 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 03:11:38.375083 22450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aa4ed319a974e9b9e3842e4137d9f14: No bootstrap required, opened a new log
I20260926 03:11:38.382570 22450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2aa4ed319a974e9b9e3842e4137d9f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa4ed319a974e9b9e3842e4137d9f14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44851 } }
I20260926 03:11:38.383059 22450 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2aa4ed319a974e9b9e3842e4137d9f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:38.383173 22450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2aa4ed319a974e9b9e3842e4137d9f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa4ed319a974e9b9e3842e4137d9f14, State: Initialized, Role: FOLLOWER
I20260926 03:11:38.383783 22450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2aa4ed319a974e9b9e3842e4137d9f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa4ed319a974e9b9e3842e4137d9f14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44851 } }
I20260926 03:11:38.384013 22450 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2aa4ed319a974e9b9e3842e4137d9f14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:11:38.384147 22450 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2aa4ed319a974e9b9e3842e4137d9f14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:11:38.384290 22450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2aa4ed319a974e9b9e3842e4137d9f14 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:38.386472 22450 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2aa4ed319a974e9b9e3842e4137d9f14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa4ed319a974e9b9e3842e4137d9f14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44851 } }
I20260926 03:11:38.387027 22450 leader_election.cc:302] T 00000000000000000000000000000000 P 2aa4ed319a974e9b9e3842e4137d9f14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2aa4ed319a974e9b9e3842e4137d9f14; no voters:
I20260926 03:11:38.387598 22450 leader_election.cc:288] T 00000000000000000000000000000000 P 2aa4ed319a974e9b9e3842e4137d9f14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:11:38.387728 22460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2aa4ed319a974e9b9e3842e4137d9f14 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:11:38.388319 22460 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2aa4ed319a974e9b9e3842e4137d9f14 [term 1 LEADER]: Becoming Leader. State: Replica: 2aa4ed319a974e9b9e3842e4137d9f14, State: Running, Role: LEADER
I20260926 03:11:38.388936 22460 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2aa4ed319a974e9b9e3842e4137d9f14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa4ed319a974e9b9e3842e4137d9f14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44851 } }
I20260926 03:11:38.389508 22450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2aa4ed319a974e9b9e3842e4137d9f14 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:11:38.391543 22461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aa4ed319a974e9b9e3842e4137d9f14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2aa4ed319a974e9b9e3842e4137d9f14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa4ed319a974e9b9e3842e4137d9f14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44851 } } }
I20260926 03:11:38.391911 22461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aa4ed319a974e9b9e3842e4137d9f14 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:38.392228 22462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aa4ed319a974e9b9e3842e4137d9f14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2aa4ed319a974e9b9e3842e4137d9f14. Latest consensus state: current_term: 1 leader_uuid: "2aa4ed319a974e9b9e3842e4137d9f14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa4ed319a974e9b9e3842e4137d9f14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44851 } } }
I20260926 03:11:38.392745 22462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aa4ed319a974e9b9e3842e4137d9f14 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:38.395565 22465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:11:38.401050 22465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:11:38.410116 22465 catalog_manager.cc:1347] Generated new cluster ID: 30b8deeae6634ab18739cfcba745801f
I20260926 03:11:38.410248 22465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:11:38.421471 22465 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:11:38.422053 22465 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:11:38.434957 22465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2aa4ed319a974e9b9e3842e4137d9f14: Generated new TSK 0
I20260926 03:11:38.435947 22465 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:11:38.448673 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 22383
W20260926 03:11:38.450409 22445 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44619
W20260926 03:11:38.451139 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 22383: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:11:38.451298 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 22383
2026-09-26T03:11:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1820 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:11:38 dist-test-slave-hmbw krb5kdc[22481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:11:38 dist-test-slave-hmbw krb5kdc[22481](info): set up 2 sockets
Sep 26 03:11:38 dist-test-slave-hmbw krb5kdc[22481](info): commencing operation
krb5kdc: starting...
W20260926 03:11:39.482736 388 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.006s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:11:39 dist-test-slave-hmbw krb5kdc[22481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:11:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:11:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:11:39.742275 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36533
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:11:40.080997 22494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:40.081347 22494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:40.081468 22494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:40.088893 22494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:11:40.089016 22494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:40.089071 22494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:40.089160 22494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:11:40.089207 22494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:11:40.099509 22494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:40.100858 22494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:40.102207 22494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:40.114238 22499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:40.114245 22500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:40.114518 22502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:40.115075 22494 server_base.cc:1034] running on GCE node
I20260926 03:11:40.115890 22494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:40.117475 22494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:40.118681 22494 hybrid_clock.cc:648] HybridClock initialized: now 1790392300118635 us; error 47 us; skew 500 ppm
I20260926 03:11:40.119061 22494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:11:40 dist-test-slave-hmbw krb5kdc[22481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:11:40.122167 22494 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:11:40.123842 22494 webserver.cc:460] Webserver started at http://127.0.97.62:40477/ using document root <none> and password file <none>
I20260926 03:11:40.124583 22494 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:40.124708 22494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:40.125007 22494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:40.128204 22494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "8a7ea1830d394610b9c8b7318cfc32d2"
format_stamp: "Formatted at 2026-09-26 03:11:40 on dist-test-slave-hmbw"
server_key: "9c86c7e0f6e27e77e542b29cb2b2b090"
server_key_iv: "7d81faf3dda145126c3def32a94a5810"
server_key_version: "encryptionkey@0"
I20260926 03:11:40.129040 22494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "8a7ea1830d394610b9c8b7318cfc32d2"
format_stamp: "Formatted at 2026-09-26 03:11:40 on dist-test-slave-hmbw"
server_key: "9c86c7e0f6e27e77e542b29cb2b2b090"
server_key_iv: "7d81faf3dda145126c3def32a94a5810"
server_key_version: "encryptionkey@0"
I20260926 03:11:40.134544 22494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 03:11:40.138108 22509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:40.139703 22494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 03:11:40.139905 22494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "8a7ea1830d394610b9c8b7318cfc32d2"
format_stamp: "Formatted at 2026-09-26 03:11:40 on dist-test-slave-hmbw"
server_key: "9c86c7e0f6e27e77e542b29cb2b2b090"
server_key_iv: "7d81faf3dda145126c3def32a94a5810"
server_key_version: "encryptionkey@0"
I20260926 03:11:40.140120 22494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:40.156045 22494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:40.160688 22494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:40.160986 22494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:40.179168 22494 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36533
I20260926 03:11:40.179178 22560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36533 every 8 connection(s)
I20260926 03:11:40.180673 22494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:11:40.184554 22561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:40.187273 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 22494
I20260926 03:11:40.187456 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:11:40.187981 388 external_mini_cluster.cc:1428] Setting key b6acedcadcc8545dcf6898b698989aba
I20260926 03:11:40.196770 22561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a7ea1830d394610b9c8b7318cfc32d2: Bootstrap starting.
I20260926 03:11:40.200121 22561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a7ea1830d394610b9c8b7318cfc32d2: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:40.201148 22561 log.cc:824] T 00000000000000000000000000000000 P 8a7ea1830d394610b9c8b7318cfc32d2: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:11:40 dist-test-slave-hmbw krb5kdc[22481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:11:40.204457 22561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a7ea1830d394610b9c8b7318cfc32d2: No bootstrap required, opened a new log
I20260926 03:11:40.211853 22564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:40.190124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49261 (local address 127.0.97.62:36533)
0926 03:11:40.191005 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:40.191018 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:40.191063 (+ 45us) server_negotiation.cc:407] Connection header received
0926 03:11:40.192793 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:40.192818 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:40.193263 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:40.193698 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:40.195228 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:40.196604 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:40.198069 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:40.198912 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:40.199092 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:40.203462 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:40.203471 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:40.203488 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:40.203532 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:40.206911 (+ 3379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:40.207912 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:40.207920 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:40.207928 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:40.208137 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:40.208631 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:40.208636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:40.208638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:40.209382 (+ 744us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:40.209655 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:40.210135 (+ 480us) server_negotiation.cc:299] Negotiation successful
0926 03:11:40.210712 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":369,"threads_started":1}
I20260926 03:11:40.213485 22561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a7ea1830d394610b9c8b7318cfc32d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7ea1830d394610b9c8b7318cfc32d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36533 } }
I20260926 03:11:40.213850 22561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a7ea1830d394610b9c8b7318cfc32d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:40.213958 22561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a7ea1830d394610b9c8b7318cfc32d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a7ea1830d394610b9c8b7318cfc32d2, State: Initialized, Role: FOLLOWER
I20260926 03:11:40.214643 22561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a7ea1830d394610b9c8b7318cfc32d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7ea1830d394610b9c8b7318cfc32d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36533 } }
I20260926 03:11:40.214933 22561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a7ea1830d394610b9c8b7318cfc32d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:11:40.215073 22561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a7ea1830d394610b9c8b7318cfc32d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:11:40.215281 22561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a7ea1830d394610b9c8b7318cfc32d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:40.215997 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:11:40.218537 22561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a7ea1830d394610b9c8b7318cfc32d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7ea1830d394610b9c8b7318cfc32d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36533 } }
I20260926 03:11:40.219252 22561 leader_election.cc:302] T 00000000000000000000000000000000 P 8a7ea1830d394610b9c8b7318cfc32d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a7ea1830d394610b9c8b7318cfc32d2; no voters:
I20260926 03:11:40.219982 22561 leader_election.cc:288] T 00000000000000000000000000000000 P 8a7ea1830d394610b9c8b7318cfc32d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:11:40.220122 22568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a7ea1830d394610b9c8b7318cfc32d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:11:40.221323 22568 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a7ea1830d394610b9c8b7318cfc32d2 [term 1 LEADER]: Becoming Leader. State: Replica: 8a7ea1830d394610b9c8b7318cfc32d2, State: Running, Role: LEADER
I20260926 03:11:40.222024 22568 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a7ea1830d394610b9c8b7318cfc32d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7ea1830d394610b9c8b7318cfc32d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36533 } }
I20260926 03:11:40.222461 22561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a7ea1830d394610b9c8b7318cfc32d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:11:40.225255 22571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a7ea1830d394610b9c8b7318cfc32d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a7ea1830d394610b9c8b7318cfc32d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7ea1830d394610b9c8b7318cfc32d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36533 } } }
I20260926 03:11:40.225636 22571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a7ea1830d394610b9c8b7318cfc32d2 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:40.226166 22572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a7ea1830d394610b9c8b7318cfc32d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a7ea1830d394610b9c8b7318cfc32d2. Latest consensus state: current_term: 1 leader_uuid: "8a7ea1830d394610b9c8b7318cfc32d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7ea1830d394610b9c8b7318cfc32d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36533 } } }
I20260926 03:11:40.226554 22572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a7ea1830d394610b9c8b7318cfc32d2 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:40.230195 22577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:11:40.236946 22577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:11:40.239699 22564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:40.221669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51527 (local address 127.0.97.62:36533)
0926 03:11:40.221860 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:40.221866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:40.222080 (+ 214us) server_negotiation.cc:407] Connection header received
0926 03:11:40.222315 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:40.222320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:40.222406 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:40.222552 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:40.224385 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:40.225532 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:40.227404 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:40.228172 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:40.228280 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:40.231568 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:40.231577 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:40.231582 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:40.231631 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:40.235558 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:40.236486 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:40.236491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:40.236495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:40.236699 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:40.237171 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:40.237176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:40.237179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:40.237591 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:40.238951 (+ 1360us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:40.239106 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 03:11:40.239334 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:11:40.248359 22577 catalog_manager.cc:1347] Generated new cluster ID: d5d59551a23a4deba803bb2985e09d61
I20260926 03:11:40.248477 22577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:11:40.269887 22577 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:11:40.271376 22577 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:11:40.279978 22577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a7ea1830d394610b9c8b7318cfc32d2: Generated new TSK 0
I20260926 03:11:40.280910 22577 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:11:40.290688 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 22494
I20260926 03:11:40.291929 22556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:11:40.384974 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 22494
2026-09-26T03:11:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1933 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:11:40 dist-test-slave-hmbw krb5kdc[22594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:11:40 dist-test-slave-hmbw krb5kdc[22594](info): set up 2 sockets
Sep 26 03:11:40 dist-test-slave-hmbw krb5kdc[22594](info): commencing operation
krb5kdc: starting...
W20260926 03:11:41.426911 388 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.007s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:11:41 dist-test-slave-hmbw krb5kdc[22594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:11:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:11:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:11:41.678458 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46285
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:11:42.017746 22607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:42.018071 22607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:42.018167 22607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:42.026014 22607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:11:42.026166 22607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:42.026216 22607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:42.026278 22607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:11:42.026324 22607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:11:42.037153 22607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:42.038546 22607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:42.039949 22607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:42.053808 22612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:42.053951 22613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:42.053838 22615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:42.055397 22607 server_base.cc:1034] running on GCE node
I20260926 03:11:42.056348 22607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:42.058048 22607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:42.059286 22607 hybrid_clock.cc:648] HybridClock initialized: now 1790392302059231 us; error 56 us; skew 500 ppm
I20260926 03:11:42.059695 22607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:11:42 dist-test-slave-hmbw krb5kdc[22594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:11:42.065295 22607 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:11:42.068023 22607 webserver.cc:460] Webserver started at http://127.0.97.62:40009/ using document root <none> and password file <none>
I20260926 03:11:42.069064 22607 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:42.069219 22607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:42.069568 22607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:42.073922 22607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "fcde2a681574436b9952886fe908dd54"
format_stamp: "Formatted at 2026-09-26 03:11:42 on dist-test-slave-hmbw"
server_key: "4e9f543e07391338ff090c7d95e23de3"
server_key_iv: "2af1f2809a87d13402d25f6ca3e6ef76"
server_key_version: "encryptionkey@0"
I20260926 03:11:42.075014 22607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "fcde2a681574436b9952886fe908dd54"
format_stamp: "Formatted at 2026-09-26 03:11:42 on dist-test-slave-hmbw"
server_key: "4e9f543e07391338ff090c7d95e23de3"
server_key_iv: "2af1f2809a87d13402d25f6ca3e6ef76"
server_key_version: "encryptionkey@0"
I20260926 03:11:42.083132 22607 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260926 03:11:42.087321 22622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:42.089051 22607 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 03:11:42.089246 22607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "fcde2a681574436b9952886fe908dd54"
format_stamp: "Formatted at 2026-09-26 03:11:42 on dist-test-slave-hmbw"
server_key: "4e9f543e07391338ff090c7d95e23de3"
server_key_iv: "2af1f2809a87d13402d25f6ca3e6ef76"
server_key_version: "encryptionkey@0"
I20260926 03:11:42.089447 22607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:42.133567 22607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:42.138010 22607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:42.138283 22607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:42.156145 22607 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46285
I20260926 03:11:42.156167 22673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46285 every 8 connection(s)
I20260926 03:11:42.157884 22607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:11:42.163044 22674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:42.164906 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 22607
I20260926 03:11:42.165048 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:11:42.165490 388 external_mini_cluster.cc:1428] Setting key 64b57e142d133912d5232657bfc817c9
I20260926 03:11:42.176860 22674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcde2a681574436b9952886fe908dd54: Bootstrap starting.
I20260926 03:11:42.180312 22674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcde2a681574436b9952886fe908dd54: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:42.181519 22674 log.cc:824] T 00000000000000000000000000000000 P fcde2a681574436b9952886fe908dd54: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:42.182482 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:42.167475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40165 (local address 127.0.97.62:46285)
0926 03:11:42.168338 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:42.168350 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:42.171232 (+ 2882us) server_negotiation.cc:407] Connection header received
0926 03:11:42.173186 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:42.173215 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:42.173700 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:42.174117 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:42.174830 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:42.176187 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:42.177570 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:42.178285 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:42.178472 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:42.179811 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:42.179820 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:42.179836 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:42.180443 (+ 607us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:42.180544 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:42.180574 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:42.180702 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:42.180938 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 03:11:42.181230 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":376,"threads_started":1}
I20260926 03:11:42.184970 22674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcde2a681574436b9952886fe908dd54: No bootstrap required, opened a new log
I20260926 03:11:42.186882 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:11:42.193512 22674 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcde2a681574436b9952886fe908dd54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcde2a681574436b9952886fe908dd54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46285 } }
I20260926 03:11:42.194046 22674 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcde2a681574436b9952886fe908dd54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:42.194170 22674 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcde2a681574436b9952886fe908dd54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcde2a681574436b9952886fe908dd54, State: Initialized, Role: FOLLOWER
I20260926 03:11:42.194823 22674 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcde2a681574436b9952886fe908dd54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcde2a681574436b9952886fe908dd54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46285 } }
I20260926 03:11:42.195057 22674 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcde2a681574436b9952886fe908dd54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:11:42.195190 22674 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcde2a681574436b9952886fe908dd54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:11:42.195365 22674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcde2a681574436b9952886fe908dd54 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:42.197695 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:42.191173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56283 (local address 127.0.97.62:46285)
0926 03:11:42.191391 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:42.191401 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:42.191754 (+ 353us) server_negotiation.cc:407] Connection header received
0926 03:11:42.191911 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:42.191919 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:42.192006 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:42.192125 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:42.192670 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:42.193712 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:42.195352 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:42.195831 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:42.195953 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:42.196987 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:42.196992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:42.196995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:42.197015 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:42.197060 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:42.197077 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:42.197154 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:42.197308 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 03:11:42.197407 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:11:42.197770 22674 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcde2a681574436b9952886fe908dd54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcde2a681574436b9952886fe908dd54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46285 } }
I20260926 03:11:42.198390 22674 leader_election.cc:302] T 00000000000000000000000000000000 P fcde2a681574436b9952886fe908dd54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcde2a681574436b9952886fe908dd54; no voters:
I20260926 03:11:42.199108 22674 leader_election.cc:288] T 00000000000000000000000000000000 P fcde2a681574436b9952886fe908dd54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:11:42.199224 22684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcde2a681574436b9952886fe908dd54 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:11:42.199826 22684 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcde2a681574436b9952886fe908dd54 [term 1 LEADER]: Becoming Leader. State: Replica: fcde2a681574436b9952886fe908dd54, State: Running, Role: LEADER
I20260926 03:11:42.200469 22684 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcde2a681574436b9952886fe908dd54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcde2a681574436b9952886fe908dd54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46285 } }
I20260926 03:11:42.201479 22674 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcde2a681574436b9952886fe908dd54 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:11:42.203397 22686 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcde2a681574436b9952886fe908dd54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcde2a681574436b9952886fe908dd54. Latest consensus state: current_term: 1 leader_uuid: "fcde2a681574436b9952886fe908dd54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcde2a681574436b9952886fe908dd54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46285 } } }
I20260926 03:11:42.203810 22686 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcde2a681574436b9952886fe908dd54 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:42.204787 22690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:11:42.205384 22685 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcde2a681574436b9952886fe908dd54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcde2a681574436b9952886fe908dd54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcde2a681574436b9952886fe908dd54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46285 } } }
I20260926 03:11:42.205799 22685 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcde2a681574436b9952886fe908dd54 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:42.215049 22690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:11:42.225337 22690 catalog_manager.cc:1347] Generated new cluster ID: ae299076857343b6a014077eeb848929
I20260926 03:11:42.225484 22690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:11:42.241092 22690 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:11:42.243105 22690 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:11:42.251171 22690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcde2a681574436b9952886fe908dd54: Generated new TSK 0
I20260926 03:11:42.252372 22690 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:11:42.258234 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 22607
W20260926 03:11:42.259863 22669 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40165
W20260926 03:11:42.260730 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 22607: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:11:42.260907 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 22607
2026-09-26T03:11:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1876 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:11:42 dist-test-slave-hmbw krb5kdc[22705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:11:42 dist-test-slave-hmbw krb5kdc[22705](info): set up 2 sockets
Sep 26 03:11:42 dist-test-slave-hmbw krb5kdc[22705](info): commencing operation
krb5kdc: starting...
W20260926 03:11:43.290823 388 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:11:43 dist-test-slave-hmbw krb5kdc[22705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:11:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:11:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:11:43.550022 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39621
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:11:43.887578 22718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:43.887888 22718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:43.887981 22718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:43.895517 22718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:11:43.895639 22718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:43.895685 22718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:43.895745 22718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:11:43.895787 22718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:11:43.906168 22718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:43.907483 22718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:43.908759 22718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:43.920859 22723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:43.920882 22726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:43.920945 22724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:43.921482 22718 server_base.cc:1034] running on GCE node
I20260926 03:11:43.922385 22718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:43.924091 22718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:43.925406 22718 hybrid_clock.cc:648] HybridClock initialized: now 1790392303925336 us; error 71 us; skew 500 ppm
I20260926 03:11:43.925809 22718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:11:43 dist-test-slave-hmbw krb5kdc[22705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:11:43.936003 22718 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:11:43.937621 22718 webserver.cc:460] Webserver started at http://127.0.97.62:40355/ using document root <none> and password file <none>
I20260926 03:11:43.938329 22718 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:43.938439 22718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:43.938771 22718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:43.941812 22718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "6b27dbcd96ff4d55ba57d10f57c8fd34"
format_stamp: "Formatted at 2026-09-26 03:11:43 on dist-test-slave-hmbw"
server_key: "4c964e5275630924b9e9b65a252c351a"
server_key_iv: "3d716297b4351e2a3e7bcf63cae32fd2"
server_key_version: "encryptionkey@0"
I20260926 03:11:43.942570 22718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "6b27dbcd96ff4d55ba57d10f57c8fd34"
format_stamp: "Formatted at 2026-09-26 03:11:43 on dist-test-slave-hmbw"
server_key: "4c964e5275630924b9e9b65a252c351a"
server_key_iv: "3d716297b4351e2a3e7bcf63cae32fd2"
server_key_version: "encryptionkey@0"
I20260926 03:11:43.948024 22718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 03:11:43.951759 22733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:43.953063 22718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:11:43.953204 22718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "6b27dbcd96ff4d55ba57d10f57c8fd34"
format_stamp: "Formatted at 2026-09-26 03:11:43 on dist-test-slave-hmbw"
server_key: "4c964e5275630924b9e9b65a252c351a"
server_key_iv: "3d716297b4351e2a3e7bcf63cae32fd2"
server_key_version: "encryptionkey@0"
I20260926 03:11:43.953379 22718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:43.959254 22718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:43.963292 22718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:43.963546 22718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:43.980691 22718 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39621
I20260926 03:11:43.980722 22784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39621 every 8 connection(s)
I20260926 03:11:43.982102 22718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:11:43.985638 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 22718
I20260926 03:11:43.985822 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:11:43.986332 22785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:43.986373 388 external_mini_cluster.cc:1428] Setting key 66bc64785f49230e93c39c700f061f30
I20260926 03:11:43.994485 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:43.988214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52633 (local address 127.0.97.62:39621)
0926 03:11:43.988989 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:43.989000 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:43.989043 (+ 43us) server_negotiation.cc:407] Connection header received
0926 03:11:43.991110 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:43.991134 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:43.991501 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:43.991873 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:43.992880 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:43.992886 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:43.992899 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:43.992916 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:43.992977 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:43.992997 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:43.993213 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:43.993359 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 03:11:43.993533 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":320,"threads_started":1}
I20260926 03:11:43.997982 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:11:43.999392 22785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b27dbcd96ff4d55ba57d10f57c8fd34: Bootstrap starting.
I20260926 03:11:44.002912 22785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b27dbcd96ff4d55ba57d10f57c8fd34: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:44.004262 22785 log.cc:824] T 00000000000000000000000000000000 P 6b27dbcd96ff4d55ba57d10f57c8fd34: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:44.004546 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:44.002099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55787 (local address 127.0.97.62:39621)
0926 03:11:44.002293 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:44.002299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:44.002652 (+ 353us) server_negotiation.cc:407] Connection header received
0926 03:11:44.002853 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:44.002857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:44.002911 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:44.003013 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:44.003937 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:44.003941 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:44.003944 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:11:44.003960 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:44.003998 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:11:44.004012 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:44.004106 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:44.004234 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 03:11:44.004285 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 03:11:44.007710 22785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b27dbcd96ff4d55ba57d10f57c8fd34: No bootstrap required, opened a new log
I20260926 03:11:44.015727 22785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b27dbcd96ff4d55ba57d10f57c8fd34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b27dbcd96ff4d55ba57d10f57c8fd34" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39621 } }
I20260926 03:11:44.016160 22785 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b27dbcd96ff4d55ba57d10f57c8fd34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:11:44.016227 22785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b27dbcd96ff4d55ba57d10f57c8fd34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b27dbcd96ff4d55ba57d10f57c8fd34, State: Initialized, Role: FOLLOWER
I20260926 03:11:44.016829 22785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b27dbcd96ff4d55ba57d10f57c8fd34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b27dbcd96ff4d55ba57d10f57c8fd34" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39621 } }
I20260926 03:11:44.017059 22785 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b27dbcd96ff4d55ba57d10f57c8fd34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:11:44.017205 22785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b27dbcd96ff4d55ba57d10f57c8fd34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:11:44.017387 22785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b27dbcd96ff4d55ba57d10f57c8fd34 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:44.019886 22785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b27dbcd96ff4d55ba57d10f57c8fd34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b27dbcd96ff4d55ba57d10f57c8fd34" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39621 } }
I20260926 03:11:44.020421 22785 leader_election.cc:302] T 00000000000000000000000000000000 P 6b27dbcd96ff4d55ba57d10f57c8fd34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b27dbcd96ff4d55ba57d10f57c8fd34; no voters:
I20260926 03:11:44.021011 22785 leader_election.cc:288] T 00000000000000000000000000000000 P 6b27dbcd96ff4d55ba57d10f57c8fd34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:11:44.021152 22795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b27dbcd96ff4d55ba57d10f57c8fd34 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:11:44.021833 22795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b27dbcd96ff4d55ba57d10f57c8fd34 [term 1 LEADER]: Becoming Leader. State: Replica: 6b27dbcd96ff4d55ba57d10f57c8fd34, State: Running, Role: LEADER
I20260926 03:11:44.022385 22795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b27dbcd96ff4d55ba57d10f57c8fd34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b27dbcd96ff4d55ba57d10f57c8fd34" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39621 } }
I20260926 03:11:44.023046 22785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b27dbcd96ff4d55ba57d10f57c8fd34 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:11:44.025154 22796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b27dbcd96ff4d55ba57d10f57c8fd34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b27dbcd96ff4d55ba57d10f57c8fd34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b27dbcd96ff4d55ba57d10f57c8fd34" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39621 } } }
I20260926 03:11:44.025017 22797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b27dbcd96ff4d55ba57d10f57c8fd34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b27dbcd96ff4d55ba57d10f57c8fd34. Latest consensus state: current_term: 1 leader_uuid: "6b27dbcd96ff4d55ba57d10f57c8fd34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b27dbcd96ff4d55ba57d10f57c8fd34" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39621 } } }
I20260926 03:11:44.025785 22797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b27dbcd96ff4d55ba57d10f57c8fd34 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:44.025785 22796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b27dbcd96ff4d55ba57d10f57c8fd34 [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:44.028327 22802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:11:44.033339 22802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:11:44.042488 22802 catalog_manager.cc:1347] Generated new cluster ID: 2c440ebcebb747558527a20571b937bc
I20260926 03:11:44.042639 22802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:11:44.055672 22802 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:11:44.056241 22802 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:11:44.065042 22802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b27dbcd96ff4d55ba57d10f57c8fd34: Generated new TSK 0
I20260926 03:11:44.066211 22802 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:11:44.074448 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 22718
W20260926 03:11:44.075583 22780 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52633
W20260926 03:11:44.076252 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 22718: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:11:44.076408 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 22718
2026-09-26T03:11:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1814 ms)
[----------] 18 tests from AuthTokenIssuingTest (33753 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:11:44 dist-test-slave-hmbw krb5kdc[22816](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:11:44 dist-test-slave-hmbw krb5kdc[22816](info): set up 2 sockets
Sep 26 03:11:44 dist-test-slave-hmbw krb5kdc[22816](info): commencing operation
krb5kdc: starting...
W20260926 03:11:45.098806 388 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:11:45 dist-test-slave-hmbw krb5kdc[22816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:11:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:11:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:11:45.359328 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34565
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34565,127.0.97.61:41111,127.0.97.60:35123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:11:45.694650 22829 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:11:45.694983 22829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:45.695077 22829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:45.695199 22829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:45.702485 22829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:11:45.702608 22829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:45.702698 22829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:45.702765 22829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:11:45.702853 22829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:11:45.713207 22829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34565,127.0.97.61:41111,127.0.97.60:35123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:45.714677 22829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:45.716018 22829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:45.727807 22837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:45.727885 22834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:45.727928 22835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:45.728682 22829 server_base.cc:1034] running on GCE node
I20260926 03:11:45.729491 22829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:45.731017 22829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:45.732308 22829 hybrid_clock.cc:648] HybridClock initialized: now 1790392305732186 us; error 44 us; skew 500 ppm
I20260926 03:11:45.732646 22829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:11:45 dist-test-slave-hmbw krb5kdc[22816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:11:45.735922 22829 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:11:45.737612 22829 webserver.cc:460] Webserver started at http://127.0.97.62:43779/ using document root <none> and password file <none>
I20260926 03:11:45.738380 22829 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:45.738493 22829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:45.738843 22829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:45.741952 22829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "038d2b7bdf38461b9f663b26183f71aa"
format_stamp: "Formatted at 2026-09-26 03:11:45 on dist-test-slave-hmbw"
server_key: "b12cc3872bfcf873ad33309e3f6e80f2"
server_key_iv: "d808c0f80d180092685e906ca1d67d61"
server_key_version: "encryptionkey@0"
I20260926 03:11:45.742844 22829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "038d2b7bdf38461b9f663b26183f71aa"
format_stamp: "Formatted at 2026-09-26 03:11:45 on dist-test-slave-hmbw"
server_key: "b12cc3872bfcf873ad33309e3f6e80f2"
server_key_iv: "d808c0f80d180092685e906ca1d67d61"
server_key_version: "encryptionkey@0"
I20260926 03:11:45.748252 22829 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:11:45.751950 22844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:45.753451 22829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:11:45.753641 22829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "038d2b7bdf38461b9f663b26183f71aa"
format_stamp: "Formatted at 2026-09-26 03:11:45 on dist-test-slave-hmbw"
server_key: "b12cc3872bfcf873ad33309e3f6e80f2"
server_key_iv: "d808c0f80d180092685e906ca1d67d61"
server_key_version: "encryptionkey@0"
I20260926 03:11:45.753861 22829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:45.789230 22829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:45.793896 22829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:45.794188 22829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:45.813431 22829 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34565
I20260926 03:11:45.813468 22895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34565 every 8 connection(s)
I20260926 03:11:45.815451 22829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:11:45.819842 22896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:45.825076 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 22829
I20260926 03:11:45.825320 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:11:45.826076 388 external_mini_cluster.cc:1428] Setting key 9b06e9ad01d6d25987191ab41544aad8
I20260926 03:11:45.827893 22896 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34565 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:11:45 dist-test-slave-hmbw krb5kdc[22816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260926 03:11:45.852958 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:45.831625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34565 (local address 127.0.0.1:43008)
0926 03:11:45.832472 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:45.832529 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:45.834970 (+ 2441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:45.835951 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:45.835959 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:45.836525 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:11:45.836867 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:45.836890 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:45.839031 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:45.839035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:45.840183 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:45.840197 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:45.841403 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:45.841409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:45.841640 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:45.842290 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:11:45.842314 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:11:45.844625 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:11:45.848696 (+ 4071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:45.848709 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:45.848719 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:45.849185 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:45.849727 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:45.849749 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:45.849751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:45.849924 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:45.850956 (+ 1032us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:11:45.850962 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:11:45.851172 (+ 210us) client_negotiation.cc:769] Sending connection context
0926 03:11:45.851391 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 03:11:45.851833 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":59,"thread_start_us":344,"threads_started":1}
I20260926 03:11:45.852998 22899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:45.832804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43008 (local address 127.0.97.62:34565)
0926 03:11:45.833448 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:45.833454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:45.833503 (+ 49us) server_negotiation.cc:407] Connection header received
0926 03:11:45.835426 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:45.835433 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:45.835696 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:45.835898 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:45.837085 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:45.838847 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:45.840340 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:45.841160 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:45.841322 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:45.844894 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:45.844905 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:45.844925 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:45.844965 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:45.848474 (+ 3509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:45.849348 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:45.849358 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:45.849362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:45.849569 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:45.850033 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:45.850038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:45.850041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:45.850800 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:45.851025 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:45.851617 (+ 592us) server_negotiation.cc:299] Negotiation successful
0926 03:11:45.852006 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":117,"server-negotiator.queue_time_us":474,"thread_start_us":306,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260926 03:11:45.856104 22896 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41111 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:11:45.857252 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:45.856781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41111 (local address 127.0.0.1:48160)
0926 03:11:45.856972 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:45.857099 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260926 03:11:45.857589 22846 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41111: connect: Connection refused (error 111)
I20260926 03:11:45.859155 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:45.858774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41111 (local address 127.0.0.1:48166)
0926 03:11:45.858908 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:45.858993 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 03:11:45.859398 22896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41111: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41111: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260926 03:11:45.886349 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:41111
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34565,127.0.97.61:41111,127.0.97.60:35123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:11:45.908763 22896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41111 } attempt: 1
I20260926 03:11:45.909935 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:45.909510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41111 (local address 127.0.0.1:48170)
0926 03:11:45.909699 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:45.909802 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 03:11:45.910899 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:45.910563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41111 (local address 127.0.0.1:48180)
0926 03:11:45.910697 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:45.910764 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 03:11:45.911099 22896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41111: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41111: connect: Connection refused (error 111)
I20260926 03:11:45.979424 22896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41111 } attempt: 2
I20260926 03:11:45.980757 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:45.980240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41111 (local address 127.0.0.1:48196)
0926 03:11:45.980520 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:45.980611 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260926 03:11:45.981921 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:45.981531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41111 (local address 127.0.0.1:48212)
0926 03:11:45.981675 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:45.981766 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 03:11:45.982133 22896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41111: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41111: connect: Connection refused (error 111)
I20260926 03:11:46.073443 22896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41111 } attempt: 3
I20260926 03:11:46.074651 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.074110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41111 (local address 127.0.0.1:48226)
0926 03:11:46.074377 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.074461 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260926 03:11:46.075680 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.075323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41111 (local address 127.0.0.1:48228)
0926 03:11:46.075468 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.075542 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 03:11:46.075881 22896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41111: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41111: connect: Connection refused (error 111)
W20260926 03:11:46.217152 22903 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:11:46.217499 22903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:46.217584 22903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:46.217667 22903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 03:11:46.219094 22896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41111 } attempt: 4
I20260926 03:11:46.220247 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.219710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41111 (local address 127.0.0.1:48244)
0926 03:11:46.220011 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.220101 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260926 03:11:46.221489 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.221005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41111 (local address 127.0.0.1:48252)
0926 03:11:46.221173 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.221330 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 03:11:46.221709 22896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41111: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41111: connect: Connection refused (error 111)
W20260926 03:11:46.225486 22903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:11:46.225584 22903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:46.225639 22903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:46.225690 22903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:11:46.225750 22903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:11:46.235996 22903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34565,127.0.97.61:41111,127.0.97.60:35123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:41111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:46.237277 22903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:46.238651 22903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:46.250883 22908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:46.250833 22909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:46.250900 22911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:46.251747 22903 server_base.cc:1034] running on GCE node
I20260926 03:11:46.252589 22903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:46.254061 22903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:46.255357 22903 hybrid_clock.cc:648] HybridClock initialized: now 1790392306255232 us; error 47 us; skew 500 ppm
I20260926 03:11:46.255739 22903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:11:46 dist-test-slave-hmbw krb5kdc[22816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:11:46.259342 22903 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260926 03:11:46.260991 22903 webserver.cc:460] Webserver started at http://127.0.97.61:35807/ using document root <none> and password file <none>
I20260926 03:11:46.261699 22903 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:46.261801 22903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:46.262076 22903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:46.265165 22903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-1/data/instance:
uuid: "f71aa9aaad5f47a4af6b1a51e289111b"
format_stamp: "Formatted at 2026-09-26 03:11:46 on dist-test-slave-hmbw"
server_key: "40742db5c7198a7b1b71f8271404bd56"
server_key_iv: "fe7fcd6f282b75ac48507eefe63cb312"
server_key_version: "encryptionkey@0"
I20260926 03:11:46.265960 22903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-1/wal/instance:
uuid: "f71aa9aaad5f47a4af6b1a51e289111b"
format_stamp: "Formatted at 2026-09-26 03:11:46 on dist-test-slave-hmbw"
server_key: "40742db5c7198a7b1b71f8271404bd56"
server_key_iv: "fe7fcd6f282b75ac48507eefe63cb312"
server_key_version: "encryptionkey@0"
I20260926 03:11:46.271564 22903 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 03:11:46.275487 22918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:46.277128 22903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 03:11:46.277320 22903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-1/wal
uuid: "f71aa9aaad5f47a4af6b1a51e289111b"
format_stamp: "Formatted at 2026-09-26 03:11:46 on dist-test-slave-hmbw"
server_key: "40742db5c7198a7b1b71f8271404bd56"
server_key_iv: "fe7fcd6f282b75ac48507eefe63cb312"
server_key_version: "encryptionkey@0"
I20260926 03:11:46.277567 22903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:46.290355 22903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:46.295020 22903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:46.295289 22903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:46.313251 22903 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:41111
I20260926 03:11:46.313313 22969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:41111 every 8 connection(s)
I20260926 03:11:46.314762 22903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-1/data/info.pb
I20260926 03:11:46.318962 22970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:46.321327 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 22903
I20260926 03:11:46.321496 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-1/wal/instance
I20260926 03:11:46.321894 388 external_mini_cluster.cc:1428] Setting key 6a5e079fed33a051315bd20d3e2e977c
I20260926 03:11:46.326359 22970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34565 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:11:46 dist-test-slave-hmbw krb5kdc[22816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260926 03:11:46.350358 22899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.329215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43018 (local address 127.0.97.62:34565)
0926 03:11:46.329437 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:46.329443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:46.330709 (+ 1266us) server_negotiation.cc:407] Connection header received
0926 03:11:46.333284 (+ 2575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:46.333289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:46.333380 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:46.333475 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:46.335715 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.336441 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.338728 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.339177 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.339292 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:46.343863 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:46.343870 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:46.343873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:46.343915 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:46.346943 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:46.347852 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:46.347857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:46.347859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:46.347994 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:46.348460 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:46.348464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:46.348466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:46.348788 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:46.348947 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:46.349830 (+ 883us) server_negotiation.cc:299] Negotiation successful
0926 03:11:46.350042 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 03:11:46.351342 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.329688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34565 (local address 127.0.0.1:43018)
0926 03:11:46.330529 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.330585 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:46.332909 (+ 2324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:46.333755 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:46.333769 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:46.334573 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:11:46.335420 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:46.335448 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.336627 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.336633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:46.338491 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:46.338545 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.339430 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.339437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:46.339747 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:46.340528 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:11:46.340559 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:11:46.343495 (+ 2936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:11:46.347136 (+ 3641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:46.347147 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:46.347164 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:46.347682 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:46.348116 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:46.348121 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:46.348124 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:46.348334 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:46.348923 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:11:46.348933 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:11:46.349416 (+ 483us) client_negotiation.cc:769] Sending connection context
0926 03:11:46.349710 (+ 294us) client_negotiation.cc:240] Negotiation successful
0926 03:11:46.350155 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":343,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260926 03:11:46.353217 22970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41111 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:11:46 dist-test-slave-hmbw krb5kdc[22816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:11:46.371212 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.354085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41111 (local address 127.0.0.1:48262)
0926 03:11:46.354271 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.354292 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:46.354418 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:46.358369 (+ 3951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:46.358374 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:46.358859 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:11:46.358997 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:46.359006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.360447 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.360451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:46.361426 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:46.361441 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.362323 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.362329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:46.362560 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:46.363139 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:11:46.363157 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:11:46.365205 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:11:46.368732 (+ 3527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:46.368737 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:46.368741 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:46.369130 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:46.369501 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:46.369506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:46.369509 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:46.369637 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:46.370245 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:11:46.370250 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:11:46.370380 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 03:11:46.370521 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 03:11:46.370825 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 03:11:46.371294 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.354106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48262 (local address 127.0.97.61:41111)
0926 03:11:46.357843 (+ 3737us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:46.357859 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:46.357892 (+ 33us) server_negotiation.cc:407] Connection header received
0926 03:11:46.357953 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:46.357959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:46.358177 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:46.358363 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:46.359175 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.360274 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.361561 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.362152 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.362269 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:46.365423 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:46.365438 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:46.365456 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:46.365490 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:46.368573 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:46.369263 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:46.369270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:46.369272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:46.369398 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:46.369732 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:46.369737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:46.369739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:46.370102 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:46.370300 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:46.370684 (+ 384us) server_negotiation.cc:299] Negotiation successful
0926 03:11:46.370941 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":3550,"thread_start_us":3379,"threads_started":1}
I20260926 03:11:46.373651 22970 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:11:46.374963 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.374446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35123 (local address 127.0.0.1:53700)
0926 03:11:46.374660 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.374787 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260926 03:11:46.375281 22919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35123: connect: Connection refused (error 111)
I20260926 03:11:46.376844 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.376480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35123 (local address 127.0.0.1:53708)
0926 03:11:46.376628 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.376718 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 03:11:46.377058 22970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35123: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35123: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260926 03:11:46.382479 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35123
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34565,127.0.97.61:41111,127.0.97.60:35123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:11:46.426390 22970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } attempt: 1
I20260926 03:11:46.427688 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.427182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35123 (local address 127.0.0.1:53720)
0926 03:11:46.427441 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.427532 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260926 03:11:46.428817 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.428426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35123 (local address 127.0.0.1:53732)
0926 03:11:46.428599 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.428692 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 03:11:46.429035 22970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35123: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35123: connect: Connection refused (error 111)
I20260926 03:11:46.497314 22970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } attempt: 2
I20260926 03:11:46.498534 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.498028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35123 (local address 127.0.0.1:53748)
0926 03:11:46.498237 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.498339 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 03:11:46.499598 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.499201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35123 (local address 127.0.0.1:53764)
0926 03:11:46.499400 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.499474 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 03:11:46.499818 22970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35123: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35123: connect: Connection refused (error 111)
I20260926 03:11:46.520964 22896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41111 } attempt: 5
Sep 26 03:11:46 dist-test-slave-hmbw krb5kdc[22816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:11:46.535746 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.521743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41111 (local address 127.0.0.1:48268)
0926 03:11:46.521959 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.521980 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:46.522089 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:46.522535 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:46.522541 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:46.522923 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:11:46.523108 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:46.523118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.524590 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.524595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:46.525467 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:46.525480 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.526546 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.526553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:46.526733 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:46.527378 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:11:46.527396 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:11:46.529482 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:11:46.533221 (+ 3739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:46.533226 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:46.533228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:46.533634 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:46.534029 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:46.534032 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:46.534034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:46.534217 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:46.534893 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:11:46.534897 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:11:46.535041 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 03:11:46.535242 (+ 201us) client_negotiation.cc:240] Negotiation successful
0926 03:11:46.535480 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 03:11:46.535979 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.521877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48268 (local address 127.0.97.61:41111)
0926 03:11:46.522086 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:46.522095 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:46.522121 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:11:46.522193 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:46.522199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:46.522318 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:46.522437 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:46.523297 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.524426 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.525631 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.526277 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.526445 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:46.529716 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:46.529722 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:46.529727 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:46.529771 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:46.533004 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:46.533760 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:46.533765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:46.533769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:46.533930 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:46.534326 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:46.534331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:46.534334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:46.534748 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:46.534889 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:46.535394 (+ 505us) server_negotiation.cc:299] Negotiation successful
0926 03:11:46.535652 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:11:46.536893 22896 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:11:46.537930 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.537544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35123 (local address 127.0.0.1:53776)
0926 03:11:46.537712 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.537789 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 03:11:46.538915 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.538598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35123 (local address 127.0.0.1:53780)
0926 03:11:46.538730 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.538814 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 03:11:46.539109 22896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35123: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35123: connect: Connection refused (error 111)
I20260926 03:11:46.590375 22896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } attempt: 1
I20260926 03:11:46.591009 22970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } attempt: 3
I20260926 03:11:46.591629 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.591114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35123 (local address 127.0.0.1:53784)
0926 03:11:46.591378 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.591472 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260926 03:11:46.592195 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.591649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35123 (local address 127.0.0.1:53786)
0926 03:11:46.591936 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.592019 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260926 03:11:46.592675 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.592265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35123 (local address 127.0.0.1:53796)
0926 03:11:46.592463 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.592540 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260926 03:11:46.592869 22896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35123: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35123: connect: Connection refused (error 111)
I20260926 03:11:46.593513 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.592771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35123 (local address 127.0.0.1:53808)
0926 03:11:46.593206 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.593297 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321}
W20260926 03:11:46.593724 22970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35123: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35123: connect: Connection refused (error 111)
I20260926 03:11:46.661108 22896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } attempt: 2
I20260926 03:11:46.662235 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.661778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35123 (local address 127.0.0.1:53810)
0926 03:11:46.661989 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.662080 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 03:11:46.663278 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.662961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35123 (local address 127.0.0.1:53826)
0926 03:11:46.663096 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.663166 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 03:11:46.663488 22896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35123: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35123: connect: Connection refused (error 111)
W20260926 03:11:46.715277 22977 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:11:46.715667 22977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:46.715741 22977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:46.715837 22977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:46.723371 22977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:11:46.723484 22977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:46.723539 22977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:46.723601 22977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:11:46.723688 22977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:11:46.734059 22977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34565,127.0.97.61:41111,127.0.97.60:35123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:46.735453 22977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:46.736797 22977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 03:11:46.736944 22970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } attempt: 4
I20260926 03:11:46.738231 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.737627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35123 (local address 127.0.0.1:53830)
0926 03:11:46.737930 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.738041 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 03:11:46.739421 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.739006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35123 (local address 127.0.0.1:53834)
0926 03:11:46.739173 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.739265 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 03:11:46.739686 22970 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35123: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35123: connect: Connection refused (error 111)
W20260926 03:11:46.751117 22982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:46.751081 22983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:46.751147 22985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:46.751899 22977 server_base.cc:1034] running on GCE node
I20260926 03:11:46.752743 22977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:46.754284 22977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:46.755553 22977 hybrid_clock.cc:648] HybridClock initialized: now 1790392306755488 us; error 66 us; skew 500 ppm
I20260926 03:11:46.755923 22977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:11:46 dist-test-slave-hmbw krb5kdc[22816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:11:46.759495 22977 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260926 03:11:46.761328 22977 webserver.cc:460] Webserver started at http://127.0.97.60:46015/ using document root <none> and password file <none>
I20260926 03:11:46.762140 22977 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:46.762259 22977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:46.762648 22977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:46.765856 22977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-2/data/instance:
uuid: "8397641d00fd415b925efbcfc93d88f7"
format_stamp: "Formatted at 2026-09-26 03:11:46 on dist-test-slave-hmbw"
server_key: "cc4d50ad2ab0a1b7f10df97b6e6b8c8d"
server_key_iv: "8e4a09b79904b9317308947ea20a91ca"
server_key_version: "encryptionkey@0"
I20260926 03:11:46.766901 22977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-2/wal/instance:
uuid: "8397641d00fd415b925efbcfc93d88f7"
format_stamp: "Formatted at 2026-09-26 03:11:46 on dist-test-slave-hmbw"
server_key: "cc4d50ad2ab0a1b7f10df97b6e6b8c8d"
server_key_iv: "8e4a09b79904b9317308947ea20a91ca"
server_key_version: "encryptionkey@0"
I20260926 03:11:46.769709 22896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } attempt: 3
I20260926 03:11:46.770939 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.770431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35123 (local address 127.0.0.1:53840)
0926 03:11:46.770689 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.770779 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 03:11:46.772059 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.771601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35123 (local address 127.0.0.1:53854)
0926 03:11:46.771808 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.771910 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260926 03:11:46.772303 22896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35123: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35123: connect: Connection refused (error 111)
I20260926 03:11:46.773435 22977 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 03:11:46.777691 22992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:46.779250 22977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:11:46.779421 22977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-2/wal
uuid: "8397641d00fd415b925efbcfc93d88f7"
format_stamp: "Formatted at 2026-09-26 03:11:46 on dist-test-slave-hmbw"
server_key: "cc4d50ad2ab0a1b7f10df97b6e6b8c8d"
server_key_iv: "8e4a09b79904b9317308947ea20a91ca"
server_key_version: "encryptionkey@0"
I20260926 03:11:46.779623 22977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:46.809787 22977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:46.814373 22977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:46.814697 22977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:46.831945 22977 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:35123
I20260926 03:11:46.831964 23043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:35123 every 8 connection(s)
I20260926 03:11:46.833302 22977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-2/data/info.pb
I20260926 03:11:46.837131 23044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:46.837973 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 22977
I20260926 03:11:46.838160 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391950058021-388-0/minicluster-data/master-2/wal/instance
I20260926 03:11:46.838721 388 external_mini_cluster.cc:1428] Setting key e6677a87009a8b9ddb27d3514441a6a7
I20260926 03:11:46.846807 23044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34565 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:11:46 dist-test-slave-hmbw krb5kdc[22816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 03:11:46 dist-test-slave-hmbw krb5kdc[22816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:11:46.862702 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.840647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53866 (local address 127.0.97.60:35123)
0926 03:11:46.841533 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:46.841546 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:46.841594 (+ 48us) server_negotiation.cc:407] Connection header received
0926 03:11:46.843475 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:46.843499 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:46.843951 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:46.844309 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:46.846114 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.847481 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.849280 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.850377 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.850576 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:46.854851 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:46.854859 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:46.854870 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:46.854913 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:46.858318 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:46.859125 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:46.859132 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:46.859139 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:46.859280 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:46.859884 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:46.859888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:46.859891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:46.860436 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:46.860734 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:46.861080 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 03:11:46.861426 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":383,"threads_started":1}
I20260926 03:11:46.864794 22899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.849709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43034 (local address 127.0.97.62:34565)
0926 03:11:46.849906 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:46.849913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:46.850568 (+ 655us) server_negotiation.cc:407] Connection header received
0926 03:11:46.850862 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:46.850867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:46.850931 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:46.851007 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:46.852503 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.853196 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.854603 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.855098 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.855193 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:46.859044 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:46.859051 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:46.859055 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:46.859098 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:46.861921 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:46.862729 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:46.862735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:46.862738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:46.862920 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:46.863386 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:46.863391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:46.863394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:46.863763 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:46.863916 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:46.864344 (+ 428us) server_negotiation.cc:299] Negotiation successful
0926 03:11:46.864534 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:11:46.864950 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.849661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34565 (local address 127.0.0.1:43034)
0926 03:11:46.850399 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.850439 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:46.850718 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:46.851099 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:46.851106 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:46.851964 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:11:46.852318 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:46.852332 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.853352 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.853358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:46.854427 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:46.854440 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.855358 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.855364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:46.855579 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:46.856229 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:11:46.856262 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:11:46.858857 (+ 2595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:11:46.862084 (+ 3227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:46.862093 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:46.862102 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:46.862532 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:46.863035 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:46.863041 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:46.863044 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:46.863253 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:46.863882 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:11:46.863887 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:11:46.864034 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 03:11:46.864201 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 03:11:46.864518 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":311,"threads_started":1}
I20260926 03:11:46.866389 23044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41111 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:11:46 dist-test-slave-hmbw krb5kdc[22816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 26 03:11:46 dist-test-slave-hmbw krb5kdc[22816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:11:46.879850 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.867120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41111 (local address 127.0.0.1:48280)
0926 03:11:46.867295 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.867312 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:46.867419 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:46.867705 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:46.867710 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:46.868126 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:11:46.868306 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:46.868318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.869260 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.869265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:46.870113 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:46.870124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.870882 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.870887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:46.871054 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:46.871610 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:11:46.871632 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:11:46.873958 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:11:46.877363 (+ 3405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:46.877368 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:46.877371 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:46.877770 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:46.878148 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:46.878152 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:46.878155 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:46.878319 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:46.878888 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:11:46.878892 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:11:46.879036 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 03:11:46.879159 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 03:11:46.879479 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 03:11:46.879875 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.867128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48280 (local address 127.0.97.61:41111)
0926 03:11:46.867300 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:46.867306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:46.867363 (+ 57us) server_negotiation.cc:407] Connection header received
0926 03:11:46.867518 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:46.867521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:46.867578 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:46.867649 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:46.868438 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.869115 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.870261 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.870739 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.870852 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:46.874209 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:46.874216 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:46.874221 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:46.874268 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:46.877206 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:46.877895 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:46.877899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:46.877902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:46.878035 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:46.878430 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:46.878434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:46.878436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:46.878757 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:46.878908 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:46.879263 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 03:11:46.879482 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:11:46.880645 22899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.866133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43042 (local address 127.0.97.62:34565)
0926 03:11:46.866659 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:46.866662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:46.866676 (+ 14us) server_negotiation.cc:407] Connection header received
0926 03:11:46.866714 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:46.866718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:46.866776 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:46.866880 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:46.868390 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.869128 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.870407 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.870880 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.870989 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:46.875228 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:46.875234 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:46.875237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:46.875280 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:46.877932 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:46.878701 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:46.878706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:46.878708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:46.878832 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:46.879179 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:46.879182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:46.879184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:46.879471 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:46.879728 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:46.880034 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 03:11:46.880249 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410}
I20260926 03:11:46.881332 23044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:11:46 dist-test-slave-hmbw krb5kdc[22816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 03:11:46 dist-test-slave-hmbw krb5kdc[22816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:11:46.895782 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.882276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53872 (local address 127.0.97.60:35123)
0926 03:11:46.882413 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:46.882419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:46.882739 (+ 320us) server_negotiation.cc:407] Connection header received
0926 03:11:46.882818 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:46.882823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:46.882890 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:46.882959 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:46.883679 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.884499 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.886147 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.886627 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.886758 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:46.890263 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:46.890269 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:46.890274 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:46.890303 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:46.893098 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:46.893806 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:46.893811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:46.893815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:46.893961 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:46.894328 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:46.894333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:46.894337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:46.894728 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:46.894884 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:46.895241 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 03:11:46.895460 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":32}
I20260926 03:11:46.896051 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.882065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35123 (local address 127.0.0.1:53872)
0926 03:11:46.882625 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.882639 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:46.882732 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:46.882986 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:46.882990 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:46.883396 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:11:46.883557 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:46.883569 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.884647 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.884652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:46.885983 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:46.885998 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.886783 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.886788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:46.886947 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:46.887538 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:11:46.887559 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:11:46.890092 (+ 2533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:11:46.893248 (+ 3156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:46.893254 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:46.893258 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:46.893683 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:46.894075 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:46.894078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:46.894081 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:46.894221 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:46.894832 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:11:46.894836 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:11:46.894996 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 03:11:46.895156 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 03:11:46.895631 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":124}
I20260926 03:11:46.897468 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.882204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48288 (local address 127.0.97.61:41111)
0926 03:11:46.882421 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:46.882426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:46.882442 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:11:46.882490 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:46.882517 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:46.882588 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:46.882675 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:46.883895 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.885064 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.886613 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.887125 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.887244 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:46.891632 (+ 4388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:46.891637 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:46.891640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:46.891678 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:46.894801 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:46.895650 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:46.895655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:46.895658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:46.895807 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:46.896247 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:46.896253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:46.896256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:46.896529 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:46.896832 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:46.896956 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 03:11:46.897172 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260926 03:11:46.901428 23044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8397641d00fd415b925efbcfc93d88f7: Bootstrap starting.
I20260926 03:11:46.902132 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:11:46.904729 23044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8397641d00fd415b925efbcfc93d88f7: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:46.905740 23044 log.cc:824] T 00000000000000000000000000000000 P 8397641d00fd415b925efbcfc93d88f7: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:46.904683 22871 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "038d2b7bdf38461b9f663b26183f71aa"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43042
I20260926 03:11:46.908519 23044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8397641d00fd415b925efbcfc93d88f7: No bootstrap required, opened a new log
I20260926 03:11:46.915669 23044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8397641d00fd415b925efbcfc93d88f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038d2b7bdf38461b9f663b26183f71aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34565 } } peers { permanent_uuid: "f71aa9aaad5f47a4af6b1a51e289111b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41111 } } peers { permanent_uuid: "8397641d00fd415b925efbcfc93d88f7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } }
I20260926 03:11:46.916023 23044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8397641d00fd415b925efbcfc93d88f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8397641d00fd415b925efbcfc93d88f7, State: Initialized, Role: FOLLOWER
I20260926 03:11:46.916473 23044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8397641d00fd415b925efbcfc93d88f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038d2b7bdf38461b9f663b26183f71aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34565 } } peers { permanent_uuid: "f71aa9aaad5f47a4af6b1a51e289111b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41111 } } peers { permanent_uuid: "8397641d00fd415b925efbcfc93d88f7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } }
I20260926 03:11:46.917335 23050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8397641d00fd415b925efbcfc93d88f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038d2b7bdf38461b9f663b26183f71aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34565 } } peers { permanent_uuid: "f71aa9aaad5f47a4af6b1a51e289111b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41111 } } peers { permanent_uuid: "8397641d00fd415b925efbcfc93d88f7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } } }
I20260926 03:11:46.917626 23050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8397641d00fd415b925efbcfc93d88f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:11:46.918649 23044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8397641d00fd415b925efbcfc93d88f7 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 03:11:46.928434 23061 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8397641d00fd415b925efbcfc93d88f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:11:46.928593 23061 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:11:46.926211 22977 master_runner.cc:186] Error getting master registration for 127.0.97.62:34565: OK, instance_id { permanent_uuid: "038d2b7bdf38461b9f663b26183f71aa" instance_seqno: 1790392305807124 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:11:46.935679 22977 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 03:11:46.949522 22896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } attempt: 4
Sep 26 03:11:46 dist-test-slave-hmbw krb5kdc[22816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 03:11:46.966684 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.950338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35123 (local address 127.0.0.1:53888)
0926 03:11:46.950564 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:46.950582 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:46.950686 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:46.951056 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:46.951060 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:46.951577 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:11:46.951816 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:46.951830 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.953339 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.953344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:46.954882 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:46.954900 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.955970 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.955977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:46.956199 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:46.956870 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:11:46.956898 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:11:46.959672 (+ 2774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:11:46.963817 (+ 4145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:46.963824 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:46.963827 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:46.964367 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:46.964801 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:46.964805 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:46.964809 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:46.964964 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:46.965642 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:11:46.965647 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:11:46.965815 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 03:11:46.965948 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 03:11:46.966266 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 03:11:46.966714 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:46.950521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53888 (local address 127.0.97.60:35123)
0926 03:11:46.950730 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:46.950736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:46.950758 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:11:46.950822 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:46.950827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:46.950906 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:46.951020 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:46.951999 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.953188 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.955059 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:46.955758 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:46.955906 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:46.959926 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:46.959934 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:46.959939 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:46.959994 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:46.963650 (+ 3656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:46.964528 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:46.964535 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:46.964539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:46.964693 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:46.965085 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:46.965090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:46.965094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:46.965511 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:46.965672 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:46.966083 (+ 411us) server_negotiation.cc:299] Negotiation successful
0926 03:11:46.966318 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:11:46.973198 22896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 038d2b7bdf38461b9f663b26183f71aa: Bootstrap starting.
I20260926 03:11:46.977610 22896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 038d2b7bdf38461b9f663b26183f71aa: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:46.979048 22896 log.cc:824] T 00000000000000000000000000000000 P 038d2b7bdf38461b9f663b26183f71aa: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:46.982800 22896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 038d2b7bdf38461b9f663b26183f71aa: No bootstrap required, opened a new log
I20260926 03:11:46.990634 22896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 038d2b7bdf38461b9f663b26183f71aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038d2b7bdf38461b9f663b26183f71aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34565 } } peers { permanent_uuid: "f71aa9aaad5f47a4af6b1a51e289111b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41111 } } peers { permanent_uuid: "8397641d00fd415b925efbcfc93d88f7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } }
I20260926 03:11:46.991079 22896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 038d2b7bdf38461b9f663b26183f71aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038d2b7bdf38461b9f663b26183f71aa, State: Initialized, Role: FOLLOWER
I20260926 03:11:46.991619 22896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 038d2b7bdf38461b9f663b26183f71aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038d2b7bdf38461b9f663b26183f71aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34565 } } peers { permanent_uuid: "f71aa9aaad5f47a4af6b1a51e289111b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41111 } } peers { permanent_uuid: "8397641d00fd415b925efbcfc93d88f7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } }
I20260926 03:11:46.992772 23064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 038d2b7bdf38461b9f663b26183f71aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038d2b7bdf38461b9f663b26183f71aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34565 } } peers { permanent_uuid: "f71aa9aaad5f47a4af6b1a51e289111b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41111 } } peers { permanent_uuid: "8397641d00fd415b925efbcfc93d88f7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } } }
I20260926 03:11:46.993165 23064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 038d2b7bdf38461b9f663b26183f71aa [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:11:46.994113 22896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 038d2b7bdf38461b9f663b26183f71aa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:11:46.994225 22871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 038d2b7bdf38461b9f663b26183f71aa [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 03:11:46.994462 22871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 038d2b7bdf38461b9f663b26183f71aa [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:46.997200 22871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 038d2b7bdf38461b9f663b26183f71aa [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038d2b7bdf38461b9f663b26183f71aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34565 } } peers { permanent_uuid: "f71aa9aaad5f47a4af6b1a51e289111b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41111 } } peers { permanent_uuid: "8397641d00fd415b925efbcfc93d88f7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } }
I20260926 03:11:46.999473 23019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "038d2b7bdf38461b9f663b26183f71aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8397641d00fd415b925efbcfc93d88f7"
I20260926 03:11:46.999991 23019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8397641d00fd415b925efbcfc93d88f7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:47.003556 23019 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8397641d00fd415b925efbcfc93d88f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038d2b7bdf38461b9f663b26183f71aa in term 1.
I20260926 03:11:47.004012 22871 leader_election.cc:288] T 00000000000000000000000000000000 P 038d2b7bdf38461b9f663b26183f71aa [CANDIDATE]: Term 1 election: Requested vote from peers f71aa9aaad5f47a4af6b1a51e289111b (127.0.97.61:41111), 8397641d00fd415b925efbcfc93d88f7 (127.0.97.60:35123)
I20260926 03:11:47.004377 22845 leader_election.cc:302] T 00000000000000000000000000000000 P 038d2b7bdf38461b9f663b26183f71aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038d2b7bdf38461b9f663b26183f71aa, 8397641d00fd415b925efbcfc93d88f7; no voters:
I20260926 03:11:47.005106 23064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 038d2b7bdf38461b9f663b26183f71aa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:11:47.005878 23064 raft_consensus.cc:686] T 00000000000000000000000000000000 P 038d2b7bdf38461b9f663b26183f71aa [term 1 LEADER]: Becoming Leader. State: Replica: 038d2b7bdf38461b9f663b26183f71aa, State: Running, Role: LEADER
I20260926 03:11:47.006486 23064 consensus_queue.cc:237] T 00000000000000000000000000000000 P 038d2b7bdf38461b9f663b26183f71aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038d2b7bdf38461b9f663b26183f71aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34565 } } peers { permanent_uuid: "f71aa9aaad5f47a4af6b1a51e289111b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41111 } } peers { permanent_uuid: "8397641d00fd415b925efbcfc93d88f7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } }
I20260926 03:11:47.018205 23068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 038d2b7bdf38461b9f663b26183f71aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 038d2b7bdf38461b9f663b26183f71aa. Latest consensus state: current_term: 1 leader_uuid: "038d2b7bdf38461b9f663b26183f71aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038d2b7bdf38461b9f663b26183f71aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34565 } } peers { permanent_uuid: "f71aa9aaad5f47a4af6b1a51e289111b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41111 } } peers { permanent_uuid: "8397641d00fd415b925efbcfc93d88f7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } } }
I20260926 03:11:47.018782 23068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 038d2b7bdf38461b9f663b26183f71aa [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:47.031306 23072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:11:47.038079 23072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:11:47.038861 22970 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } attempt: 5
W20260926 03:11:47.047231 22850 tablet.cc:2369] T 00000000000000000000000000000000 P 038d2b7bdf38461b9f663b26183f71aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:11:47.059552 23019 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8397641d00fd415b925efbcfc93d88f7 [term 1 FOLLOWER]: Refusing update from remote peer 038d2b7bdf38461b9f663b26183f71aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:11:47.061584 23064 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 038d2b7bdf38461b9f663b26183f71aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8397641d00fd415b925efbcfc93d88f7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 03:11:47.066955 22998 tablet.cc:2369] T 00000000000000000000000000000000 P 8397641d00fd415b925efbcfc93d88f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 03:11:47 dist-test-slave-hmbw krb5kdc[22816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 03:11:47.083748 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:47.039636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35123 (local address 127.0.0.1:53906)
0926 03:11:47.039934 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:47.039957 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:47.040089 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:47.056177 (+ 16088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:47.056183 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:47.056727 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:11:47.056935 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:47.056950 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:47.058948 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:47.058954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:47.060295 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:47.060315 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:47.066673 (+ 6358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:47.066681 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:47.066899 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:47.067585 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:11:47.067610 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:11:47.070636 (+ 3026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:11:47.075029 (+ 4393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:47.075036 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:47.075040 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:47.075614 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:47.076131 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:47.076136 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:47.076139 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:47.076326 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:47.082624 (+ 6298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:11:47.082631 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:11:47.082844 (+ 213us) client_negotiation.cc:769] Sending connection context
0926 03:11:47.083012 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 03:11:47.083345 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260926 03:11:47.085325 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:47.040488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53906 (local address 127.0.97.60:35123)
0926 03:11:47.051428 (+ 10940us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:47.051435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:47.051472 (+ 37us) server_negotiation.cc:407] Connection header received
0926 03:11:47.051555 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:47.051561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:47.051671 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:47.057104 (+ 5433us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:47.057402 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:47.058753 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:47.060519 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:47.061174 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:47.067265 (+ 6091us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:47.070927 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:47.070934 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:47.070939 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:47.070985 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:47.074836 (+ 3851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:47.075798 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:47.075803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:47.075807 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:47.075989 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:47.076464 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:47.076468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:47.076471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:47.076828 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:47.084476 (+ 7648us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:47.084632 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 03:11:47.084898 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10762,"thread_start_us":10598,"threads_started":1}
I20260926 03:11:47.094201 22970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f71aa9aaad5f47a4af6b1a51e289111b: Bootstrap starting.
I20260926 03:11:47.108625 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:47.033078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48294 (local address 127.0.97.61:41111)
0926 03:11:47.033335 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:47.033343 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:47.054194 (+ 20851us) server_negotiation.cc:407] Connection header received
0926 03:11:47.054528 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:47.054535 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:47.054630 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:47.054710 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:47.065866 (+ 11156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:47.066967 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:47.075473 (+ 8506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:47.076150 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:47.076250 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:47.087763 (+ 11513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:47.087770 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:47.087775 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:47.087827 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:47.091744 (+ 3917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:47.102634 (+ 10890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:47.102642 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:47.102646 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:47.102854 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:47.103999 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:47.104005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:47.104009 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:47.104454 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:47.104870 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:47.107640 (+ 2770us) server_negotiation.cc:299] Negotiation successful
0926 03:11:47.108105 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:11:47.112614 22899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:47.034025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43056 (local address 127.0.97.62:34565)
0926 03:11:47.034229 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:47.034239 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:47.059702 (+ 25463us) server_negotiation.cc:407] Connection header received
0926 03:11:47.060884 (+ 1182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:47.060891 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:47.060985 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:47.061059 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:47.069040 (+ 7981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:47.070148 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:47.077237 (+ 7089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:47.078002 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:47.078109 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:47.092846 (+ 14737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:47.092855 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:47.092860 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:47.092919 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:47.096907 (+ 3988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:47.105550 (+ 8643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:47.105557 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:47.105561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:47.105780 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:47.106281 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:47.106286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:47.106289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:47.107716 (+ 1427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:47.108704 (+ 988us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:47.110150 (+ 1446us) server_negotiation.cc:299] Negotiation successful
0926 03:11:47.111162 (+ 1012us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 03:11:47.114437 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:47.036882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53890 (local address 127.0.97.60:35123)
0926 03:11:47.037110 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:47.037116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:47.060768 (+ 23652us) server_negotiation.cc:407] Connection header received
0926 03:11:47.062274 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:47.062280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:47.062360 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:47.062425 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:47.072059 (+ 9634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:47.073222 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:47.081472 (+ 8250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:47.082218 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:47.082327 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:47.098062 (+ 15735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:47.098071 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:47.098076 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:47.098135 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:47.102095 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:47.106310 (+ 4215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:47.106316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:47.106319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:47.106550 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:47.110457 (+ 3907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:47.110467 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:47.110471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:47.111146 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:47.111696 (+ 550us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:47.113670 (+ 1974us) server_negotiation.cc:299] Negotiation successful
0926 03:11:47.113996 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 03:11:47.116626 22970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f71aa9aaad5f47a4af6b1a51e289111b: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:47.122576 23050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8397641d00fd415b925efbcfc93d88f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 038d2b7bdf38461b9f663b26183f71aa. Latest consensus state: current_term: 1 leader_uuid: "038d2b7bdf38461b9f663b26183f71aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038d2b7bdf38461b9f663b26183f71aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34565 } } peers { permanent_uuid: "f71aa9aaad5f47a4af6b1a51e289111b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41111 } } peers { permanent_uuid: "8397641d00fd415b925efbcfc93d88f7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } } }
I20260926 03:11:47.122923 23050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8397641d00fd415b925efbcfc93d88f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:11:47.124922 23083 mvcc.cc:205] Tried to move back new op lower bound from 7333446889697361920 to 7333446889504718848. Current Snapshot: MvccSnapshot[applied={T|T < 7333446889697361920}]
I20260926 03:11:47.126551 23085 mvcc.cc:205] Tried to move back new op lower bound from 7333446889697361920 to 7333446889504718848. Current Snapshot: MvccSnapshot[applied={T|T < 7333446889697361920}]
I20260926 03:11:47.118445 22970 log.cc:824] T 00000000000000000000000000000000 P f71aa9aaad5f47a4af6b1a51e289111b: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:47.146795 23050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8397641d00fd415b925efbcfc93d88f7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "038d2b7bdf38461b9f663b26183f71aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038d2b7bdf38461b9f663b26183f71aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34565 } } peers { permanent_uuid: "f71aa9aaad5f47a4af6b1a51e289111b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41111 } } peers { permanent_uuid: "8397641d00fd415b925efbcfc93d88f7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } } }
I20260926 03:11:47.147101 23050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8397641d00fd415b925efbcfc93d88f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:11:47.147140 23068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 038d2b7bdf38461b9f663b26183f71aa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "038d2b7bdf38461b9f663b26183f71aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038d2b7bdf38461b9f663b26183f71aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34565 } } peers { permanent_uuid: "f71aa9aaad5f47a4af6b1a51e289111b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41111 } } peers { permanent_uuid: "8397641d00fd415b925efbcfc93d88f7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } } }
I20260926 03:11:47.147778 23068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 038d2b7bdf38461b9f663b26183f71aa [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:47.152781 22970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f71aa9aaad5f47a4af6b1a51e289111b: No bootstrap required, opened a new log
I20260926 03:11:47.155570 23072 catalog_manager.cc:1347] Generated new cluster ID: b75bfd0dfb3c4d7ebd1a8e1b565465a5
I20260926 03:11:47.155695 23072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:11:47.164211 22970 raft_consensus.cc:348] T 00000000000000000000000000000000 P f71aa9aaad5f47a4af6b1a51e289111b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038d2b7bdf38461b9f663b26183f71aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34565 } } peers { permanent_uuid: "f71aa9aaad5f47a4af6b1a51e289111b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41111 } } peers { permanent_uuid: "8397641d00fd415b925efbcfc93d88f7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } }
I20260926 03:11:47.164628 22970 raft_consensus.cc:729] T 00000000000000000000000000000000 P f71aa9aaad5f47a4af6b1a51e289111b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f71aa9aaad5f47a4af6b1a51e289111b, State: Initialized, Role: FOLLOWER
I20260926 03:11:47.165153 22970 consensus_queue.cc:260] T 00000000000000000000000000000000 P f71aa9aaad5f47a4af6b1a51e289111b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038d2b7bdf38461b9f663b26183f71aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34565 } } peers { permanent_uuid: "f71aa9aaad5f47a4af6b1a51e289111b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41111 } } peers { permanent_uuid: "8397641d00fd415b925efbcfc93d88f7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } }
I20260926 03:11:47.161041 22829 master_runner.cc:186] Error getting master registration for 127.0.97.61:41111: OK, instance_id { permanent_uuid: "f71aa9aaad5f47a4af6b1a51e289111b" instance_seqno: 1790392306308166 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:11:47.179404 22829 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 03:11:47.209363 23094 sys_catalog.cc:455] T 00000000000000000000000000000000 P f71aa9aaad5f47a4af6b1a51e289111b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038d2b7bdf38461b9f663b26183f71aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34565 } } peers { permanent_uuid: "f71aa9aaad5f47a4af6b1a51e289111b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41111 } } peers { permanent_uuid: "8397641d00fd415b925efbcfc93d88f7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } } }
I20260926 03:11:47.209883 23094 sys_catalog.cc:458] T 00000000000000000000000000000000 P f71aa9aaad5f47a4af6b1a51e289111b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:11:47.210836 22970 sys_catalog.cc:564] T 00000000000000000000000000000000 P f71aa9aaad5f47a4af6b1a51e289111b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:11:47.210907 22945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "038d2b7bdf38461b9f663b26183f71aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f71aa9aaad5f47a4af6b1a51e289111b"
I20260926 03:11:47.211440 22945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f71aa9aaad5f47a4af6b1a51e289111b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:47.215826 22945 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f71aa9aaad5f47a4af6b1a51e289111b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038d2b7bdf38461b9f663b26183f71aa in term 1.
I20260926 03:11:47.216838 22944 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f71aa9aaad5f47a4af6b1a51e289111b [term 1 FOLLOWER]: Refusing update from remote peer 038d2b7bdf38461b9f663b26183f71aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:11:47.222895 23068 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 038d2b7bdf38461b9f663b26183f71aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71aa9aaad5f47a4af6b1a51e289111b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:11:47.242571 23072 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:11:47.246407 23072 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:11:47.269264 23094 sys_catalog.cc:455] T 00000000000000000000000000000000 P f71aa9aaad5f47a4af6b1a51e289111b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 038d2b7bdf38461b9f663b26183f71aa. Latest consensus state: current_term: 1 leader_uuid: "038d2b7bdf38461b9f663b26183f71aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038d2b7bdf38461b9f663b26183f71aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34565 } } peers { permanent_uuid: "f71aa9aaad5f47a4af6b1a51e289111b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41111 } } peers { permanent_uuid: "8397641d00fd415b925efbcfc93d88f7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } } }
I20260926 03:11:47.269670 23094 sys_catalog.cc:458] T 00000000000000000000000000000000 P f71aa9aaad5f47a4af6b1a51e289111b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:11:47.269550 23064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 038d2b7bdf38461b9f663b26183f71aa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "038d2b7bdf38461b9f663b26183f71aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038d2b7bdf38461b9f663b26183f71aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34565 } } peers { permanent_uuid: "f71aa9aaad5f47a4af6b1a51e289111b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41111 } } peers { permanent_uuid: "8397641d00fd415b925efbcfc93d88f7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } } }
I20260926 03:11:47.269882 23064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 038d2b7bdf38461b9f663b26183f71aa [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:47.272583 23101 sys_catalog.cc:455] T 00000000000000000000000000000000 P f71aa9aaad5f47a4af6b1a51e289111b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "038d2b7bdf38461b9f663b26183f71aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038d2b7bdf38461b9f663b26183f71aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34565 } } peers { permanent_uuid: "f71aa9aaad5f47a4af6b1a51e289111b" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41111 } } peers { permanent_uuid: "8397641d00fd415b925efbcfc93d88f7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35123 } } }
I20260926 03:11:47.274042 23101 sys_catalog.cc:458] T 00000000000000000000000000000000 P f71aa9aaad5f47a4af6b1a51e289111b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:11:47.278900 23098 mvcc.cc:205] Tried to move back new op lower bound from 7333446889697361920 to 7333446889504718848. Current Snapshot: MvccSnapshot[applied={T|T < 7333446889697361920 or (T in {7333446889697361920})}]
I20260926 03:11:47.299243 23072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 038d2b7bdf38461b9f663b26183f71aa: Generated new TSK 0
I20260926 03:11:47.300400 23072 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:11:47.337116 23110 catalog_manager.cc:1259] Loaded cluster ID: b75bfd0dfb3c4d7ebd1a8e1b565465a5
I20260926 03:11:47.337337 23110 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f71aa9aaad5f47a4af6b1a51e289111b: loading cluster ID for follower catalog manager: success
I20260926 03:11:47.341336 23110 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f71aa9aaad5f47a4af6b1a51e289111b: acquiring CA information for follower catalog manager: success
I20260926 03:11:47.344815 23110 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f71aa9aaad5f47a4af6b1a51e289111b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 03:11:47.471948 22899 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:11:47.437577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43058 (local address 127.0.97.62:34565)
0926 03:11:47.437803 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:47.437809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:47.467913 (+ 30104us) server_negotiation.cc:407] Connection header received
0926 03:11:47.468159 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:47.468165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:47.468255 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:47.468378 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:47.471653 (+ 3275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43058: BlockingRecv error: recv got EOF from 127.0.0.1:43058 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 03:11:47.474592 22974 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:11:47.439599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48304 (local address 127.0.97.61:41111)
0926 03:11:47.439820 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:47.439826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:47.470365 (+ 30539us) server_negotiation.cc:407] Connection header received
0926 03:11:47.472942 (+ 2577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:47.472948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:47.473050 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:47.473131 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:47.474308 (+ 1177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48304: BlockingRecv error: recv got EOF from 127.0.0.1:48304 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260926 03:11:47.477092 23047 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:11:47.449906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53914 (local address 127.0.97.60:35123)
0926 03:11:47.450113 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:47.450120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:47.475426 (+ 25306us) server_negotiation.cc:407] Connection header received
0926 03:11:47.475670 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:47.475676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:47.475768 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:47.475855 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:47.476815 (+ 960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53914: BlockingRecv error: recv got EOF from 127.0.0.1:53914 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 03:11:47.531541 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:47.512682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48310 (local address 127.0.97.61:41111)
0926 03:11:47.512870 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:47.512876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:47.523510 (+ 10634us) server_negotiation.cc:407] Connection header received
0926 03:11:47.524062 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:47.524069 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:47.524258 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:47.524399 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:11:47.525045 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:47.526191 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:47.528411 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:47.529114 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:47.529206 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:47.529999 (+ 793us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:11:47.530753 (+ 754us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:11:47.530879 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:47.531101 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 03:11:47.531210 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 03:11:47.552412 23047 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:11:47.550294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53926 (local address 127.0.97.60:35123)
0926 03:11:47.550535 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:47.550542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:47.551054 (+ 512us) server_negotiation.cc:407] Connection header received
0926 03:11:47.551272 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:47.551276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:47.551384 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:47.551470 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:47.552184 (+ 714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53926: BlockingRecv error: recv got EOF from 127.0.0.1:53926 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 03:11:47.561786 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:47.556214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48318 (local address 127.0.97.61:41111)
0926 03:11:47.556431 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:47.556438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:47.556885 (+ 447us) server_negotiation.cc:407] Connection header received
0926 03:11:47.557130 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:47.557134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:47.557202 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:47.557276 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:11:47.557725 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:47.558694 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:47.560062 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:47.560522 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:47.560630 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:47.561042 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:11:47.561186 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:11:47.561259 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:47.561407 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 03:11:47.561490 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 03:11:47.570034 23047 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:11:47.568104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53928 (local address 127.0.97.60:35123)
0926 03:11:47.568304 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:47.568311 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:47.568775 (+ 464us) server_negotiation.cc:407] Connection header received
0926 03:11:47.568956 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:47.568962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:47.569036 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:47.569117 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:47.569800 (+ 683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53928: BlockingRecv error: recv got EOF from 127.0.0.1:53928 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:11:47.588982 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:47.583261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48326 (local address 127.0.97.61:41111)
0926 03:11:47.583498 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:47.583505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:47.584068 (+ 563us) server_negotiation.cc:407] Connection header received
0926 03:11:47.584276 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:47.584282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:47.584371 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:47.584466 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:11:47.585023 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:47.585774 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:47.587260 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:47.587695 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:47.587809 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:47.588215 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:11:47.588330 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:11:47.588405 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:47.588543 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 03:11:47.588624 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260926 03:11:47.600248 23047 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:11:47.595140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53930 (local address 127.0.97.60:35123)
0926 03:11:47.595349 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:47.595356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:47.598988 (+ 3632us) server_negotiation.cc:407] Connection header received
0926 03:11:47.599170 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:47.599176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:47.599254 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:47.599335 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:47.600036 (+ 701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53930: BlockingRecv error: recv got EOF from 127.0.0.1:53930 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:11:47.612615 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:47.607116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48342 (local address 127.0.97.61:41111)
0926 03:11:47.607397 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:47.607404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:47.607899 (+ 495us) server_negotiation.cc:407] Connection header received
0926 03:11:47.608087 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:47.608093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:47.608167 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:47.608241 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:11:47.608642 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:47.609397 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:47.610843 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:47.611351 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:47.611463 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:47.611890 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:11:47.611998 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:11:47.612077 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:47.612239 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 03:11:47.612329 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
W20260926 03:11:47.624411 23047 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:11:47.618458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53944 (local address 127.0.97.60:35123)
0926 03:11:47.618741 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:47.618747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:47.623033 (+ 4286us) server_negotiation.cc:407] Connection header received
0926 03:11:47.623225 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:47.623229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:47.623325 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:47.623412 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:47.624170 (+ 758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53944: BlockingRecv error: recv got EOF from 127.0.0.1:53944 (error 108)
Metrics: {"server-negotiator.queue_time_us":151}
I20260926 03:11:47.646018 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:47.640383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48354 (local address 127.0.97.61:41111)
0926 03:11:47.640602 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:47.640610 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:47.641108 (+ 498us) server_negotiation.cc:407] Connection header received
0926 03:11:47.641300 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:47.641304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:47.641385 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:47.641463 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:11:47.641926 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:47.642717 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:47.644053 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:47.644541 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:47.644650 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:47.645129 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:11:47.645277 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:11:47.645380 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:47.645590 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 03:11:47.645687 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260926 03:11:47.659607 23047 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:11:47.657379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53956 (local address 127.0.97.60:35123)
0926 03:11:47.657594 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:47.657601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:47.658108 (+ 507us) server_negotiation.cc:407] Connection header received
0926 03:11:47.658294 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:47.658300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:47.658405 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:47.658490 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:47.659383 (+ 893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53956: BlockingRecv error: recv got EOF from 127.0.0.1:53956 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 03:11:47.687098 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:47.679506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48362 (local address 127.0.97.61:41111)
0926 03:11:47.679741 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:47.679748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:47.680217 (+ 469us) server_negotiation.cc:407] Connection header received
0926 03:11:47.680422 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:47.680427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:47.680498 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:47.680581 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:11:47.681032 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:47.682019 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:47.684521 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:47.685025 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:47.685394 (+ 369us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:47.686046 (+ 652us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:11:47.686227 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:11:47.686322 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:47.686645 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 03:11:47.686790 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260926 03:11:47.698485 23047 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:11:47.696408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53966 (local address 127.0.97.60:35123)
0926 03:11:47.696617 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:47.696624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:47.697129 (+ 505us) server_negotiation.cc:407] Connection header received
0926 03:11:47.697327 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:47.697333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:47.697417 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:47.697501 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:47.698232 (+ 731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53966: BlockingRecv error: recv got EOF from 127.0.0.1:53966 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 03:11:47.804525 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:47.792177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48366 (local address 127.0.97.61:41111)
0926 03:11:47.792402 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:47.792409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:47.797148 (+ 4739us) server_negotiation.cc:407] Connection header received
0926 03:11:47.797505 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:47.797511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:47.797614 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:47.797686 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:11:47.798344 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:47.799470 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:47.801749 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:47.802532 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:47.802643 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:47.803620 (+ 977us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:11:47.803794 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:11:47.803860 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:47.804079 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 03:11:47.804192 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 03:11:47.844848 23047 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:11:47.832392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53976 (local address 127.0.97.60:35123)
0926 03:11:47.832579 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:47.832586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:47.843134 (+ 10548us) server_negotiation.cc:407] Connection header received
0926 03:11:47.843370 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:47.843376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:47.843477 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:47.843555 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:47.844597 (+ 1042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53976: BlockingRecv error: recv got EOF from 127.0.0.1:53976 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:11:47.918000 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:47.912961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48372 (local address 127.0.97.61:41111)
0926 03:11:47.913141 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:47.913148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:47.913579 (+ 431us) server_negotiation.cc:407] Connection header received
0926 03:11:47.913765 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:47.913769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:47.913847 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:47.913925 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:11:47.914336 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:47.915120 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:47.916425 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:47.916843 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:47.916960 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:47.917349 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:11:47.917453 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:11:47.917526 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:47.917665 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 03:11:47.917744 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 03:11:47.926935 23047 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:11:47.924944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53988 (local address 127.0.97.60:35123)
0926 03:11:47.925155 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:47.925162 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:47.925451 (+ 289us) server_negotiation.cc:407] Connection header received
0926 03:11:47.925623 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:47.925629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:47.925743 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:47.925865 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:47.926761 (+ 896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53988: BlockingRecv error: recv got EOF from 127.0.0.1:53988 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:11:47.931001 23061 catalog_manager.cc:1259] Loaded cluster ID: b75bfd0dfb3c4d7ebd1a8e1b565465a5
I20260926 03:11:47.931113 23061 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8397641d00fd415b925efbcfc93d88f7: loading cluster ID for follower catalog manager: success
I20260926 03:11:47.934815 23061 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8397641d00fd415b925efbcfc93d88f7: acquiring CA information for follower catalog manager: success
I20260926 03:11:47.937384 23061 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8397641d00fd415b925efbcfc93d88f7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:11:48.064110 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:48.059143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48384 (local address 127.0.97.61:41111)
0926 03:11:48.059412 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:48.059419 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:48.059603 (+ 184us) server_negotiation.cc:407] Connection header received
0926 03:11:48.059820 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:48.059827 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:48.059902 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:48.059980 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:11:48.060385 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:48.061095 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:48.062395 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:48.062878 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:48.062990 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:48.063400 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:11:48.063509 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:11:48.063583 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:48.063724 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 03:11:48.063807 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:11:48.075281 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:48.069630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54002 (local address 127.0.97.60:35123)
0926 03:11:48.069806 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:48.069813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:48.070114 (+ 301us) server_negotiation.cc:407] Connection header received
0926 03:11:48.070310 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:48.070315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:48.070465 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:48.070590 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:11:48.071017 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:48.071692 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:48.073017 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:48.073443 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:48.073552 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:48.074063 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:11:48.074680 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:11:48.074814 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:48.074919 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 03:11:48.075010 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 03:11:48.099495 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:48.082221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48392 (local address 127.0.97.61:41111)
0926 03:11:48.082393 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:48.082399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:48.093126 (+ 10727us) server_negotiation.cc:407] Connection header received
0926 03:11:48.093387 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:48.093395 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:48.093500 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:48.093573 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:11:48.094463 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:48.095394 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:48.097428 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:48.098041 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:48.098176 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:48.098695 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:11:48.098823 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:11:48.098890 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:48.099099 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 03:11:48.099187 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 03:11:48.100113 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:48.081863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54008 (local address 127.0.97.60:35123)
0926 03:11:48.082059 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:48.082066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:48.093930 (+ 11864us) server_negotiation.cc:407] Connection header received
0926 03:11:48.094072 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:48.094078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:48.094194 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:48.094284 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:11:48.095035 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:48.095997 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:48.097891 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:48.098437 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:48.098616 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:48.099079 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:11:48.099200 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:11:48.099282 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:48.099474 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 03:11:48.099556 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 03:11:48.100840 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:48.081729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43064 (local address 127.0.97.62:34565)
0926 03:11:48.094537 (+ 12808us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:48.094542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:48.094562 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:11:48.094680 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:48.094686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:48.094933 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:48.095125 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:11:48.095701 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:48.096401 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:48.098222 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:48.098736 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:48.098869 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:48.099426 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:11:48.099866 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:11:48.099990 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:48.100488 (+ 498us) server_negotiation.cc:299] Negotiation successful
0926 03:11:48.100578 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12650,"thread_start_us":12544,"threads_started":1}
I20260926 03:11:48.105810 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 22829
I20260926 03:11:48.106894 22891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:11:48.220880 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 22829
W20260926 03:11:48.238389 22996 connection.cc:535] server connection from 127.0.0.1:53888 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 03:11:48.238700 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 22903
I20260926 03:11:48.239742 22965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:11:48.346366 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 22903
I20260926 03:11:48.362879 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 22977
I20260926 03:11:48.363981 23039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:11:48.470362 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 22977
2026-09-26T03:11:48Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4398 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:11:48 dist-test-slave-hmbw krb5kdc[23224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:11:48 dist-test-slave-hmbw krb5kdc[23224](info): set up 2 sockets
Sep 26 03:11:48 dist-test-slave-hmbw krb5kdc[23224](info): commencing operation
krb5kdc: starting...
W20260926 03:11:49.502970 388 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/krb5kdc/test-user.keytab.
Sep 26 03:11:49 dist-test-slave-hmbw krb5kdc[23224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:11:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:11:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:11:49.761220 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35105
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35105,127.0.97.61:43193,127.0.97.60:45059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:11:50.094457 23237 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:11:50.094851 23237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:50.094945 23237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:50.095055 23237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:50.102789 23237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:11:50.102909 23237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:50.102990 23237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:50.103065 23237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:11:50.103156 23237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:11:50.113957 23237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35105,127.0.97.61:43193,127.0.97.60:45059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:50.115674 23237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:50.117137 23237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:50.129184 23245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:50.129241 23243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:50.129211 23242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:50.129993 23237 server_base.cc:1034] running on GCE node
I20260926 03:11:50.130864 23237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:50.132362 23237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:50.133620 23237 hybrid_clock.cc:648] HybridClock initialized: now 1790392310133548 us; error 43 us; skew 500 ppm
I20260926 03:11:50.133985 23237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:11:50 dist-test-slave-hmbw krb5kdc[23224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:11:50.145412 23237 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:11:50.147130 23237 webserver.cc:460] Webserver started at http://127.0.97.62:42213/ using document root <none> and password file <none>
I20260926 03:11:50.147871 23237 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:50.147965 23237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:50.148278 23237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:50.151404 23237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-0/data/instance:
uuid: "e7f16928ee304c48ab7c7282b066c89a"
format_stamp: "Formatted at 2026-09-26 03:11:50 on dist-test-slave-hmbw"
server_key: "2120886603287e29db427fcd62e86894"
server_key_iv: "f01165fdafe176f37fb59ab4a0058bbd"
server_key_version: "encryptionkey@0"
I20260926 03:11:50.152233 23237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-0/wal/instance:
uuid: "e7f16928ee304c48ab7c7282b066c89a"
format_stamp: "Formatted at 2026-09-26 03:11:50 on dist-test-slave-hmbw"
server_key: "2120886603287e29db427fcd62e86894"
server_key_iv: "f01165fdafe176f37fb59ab4a0058bbd"
server_key_version: "encryptionkey@0"
I20260926 03:11:50.157573 23237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:11:50.161068 23252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:50.162446 23237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:11:50.162654 23237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-0/wal
uuid: "e7f16928ee304c48ab7c7282b066c89a"
format_stamp: "Formatted at 2026-09-26 03:11:50 on dist-test-slave-hmbw"
server_key: "2120886603287e29db427fcd62e86894"
server_key_iv: "f01165fdafe176f37fb59ab4a0058bbd"
server_key_version: "encryptionkey@0"
I20260926 03:11:50.162864 23237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:50.175048 23237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:50.179106 23237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:50.179364 23237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:50.197552 23237 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35105
I20260926 03:11:50.197572 23303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35105 every 8 connection(s)
I20260926 03:11:50.198982 23237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:11:50.202716 23304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:50.206678 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 23237
I20260926 03:11:50.206837 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-0/wal/instance
I20260926 03:11:50.207283 388 external_mini_cluster.cc:1428] Setting key 0b0aa24c29025403f16855e748c242be
I20260926 03:11:50.209487 23304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:11:50 dist-test-slave-hmbw krb5kdc[23224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260926 03:11:50.234671 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.212805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35105 (local address 127.0.0.1:56526)
0926 03:11:50.215089 (+ 2284us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:50.215141 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:50.217545 (+ 2404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:50.218565 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:50.218573 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:50.219329 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:11:50.219759 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:50.219781 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:50.221384 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:50.221389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:50.222997 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:50.223013 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:50.224057 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:50.224065 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:50.224217 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:50.224867 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:11:50.224899 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:11:50.227413 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:11:50.230574 (+ 3161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:50.230586 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:50.230599 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:50.230995 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:50.231448 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:50.231453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:50.231457 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:50.231653 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:50.232499 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:11:50.232506 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:11:50.232700 (+ 194us) client_negotiation.cc:769] Sending connection context
0926 03:11:50.232924 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 03:11:50.233556 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1999,"mutex_wait_us":190,"spinlock_wait_cycles":15616,"thread_start_us":1748,"threads_started":1}
I20260926 03:11:50.234726 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.212805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56526 (local address 127.0.97.62:35105)
0926 03:11:50.213704 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:50.213720 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:50.215299 (+ 1579us) server_negotiation.cc:407] Connection header received
0926 03:11:50.218011 (+ 2712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:50.218017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:50.218284 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:50.218462 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:50.219984 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:50.221218 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:50.223150 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:50.223753 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:50.223929 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:50.227648 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:50.227656 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:50.227670 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:50.227704 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:50.230404 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:50.231131 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:50.231138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:50.231140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:50.231307 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:50.231769 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:50.231772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:50.231774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:50.232363 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:50.232537 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:50.233074 (+ 537us) server_negotiation.cc:299] Negotiation successful
0926 03:11:50.233410 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":406,"threads_started":1}
I20260926 03:11:50.237635 23304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43193 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:11:50.238651 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.238208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43193 (local address 127.0.0.1:56396)
0926 03:11:50.238383 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:50.238489 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 03:11:50.238901 23255 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43193: connect: Connection refused (error 111)
I20260926 03:11:50.240383 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.240035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43193 (local address 127.0.0.1:56402)
0926 03:11:50.240171 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:50.240252 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 03:11:50.240576 23304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43193: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43193: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260926 03:11:50.265882 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:43193
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35105,127.0.97.61:43193,127.0.97.60:45059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:11:50.289927 23304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43193 } attempt: 1
I20260926 03:11:50.291049 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.290631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43193 (local address 127.0.0.1:56414)
0926 03:11:50.290824 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:50.290921 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 03:11:50.292085 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.291741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43193 (local address 127.0.0.1:56418)
0926 03:11:50.291890 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:50.291961 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 03:11:50.292287 23304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43193: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43193: connect: Connection refused (error 111)
I20260926 03:11:50.360564 23304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43193 } attempt: 2
I20260926 03:11:50.361688 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.361234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43193 (local address 127.0.0.1:56424)
0926 03:11:50.361467 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:50.361556 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 03:11:50.362888 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.362431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43193 (local address 127.0.0.1:56436)
0926 03:11:50.362654 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:50.362736 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260926 03:11:50.363116 23304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43193: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43193: connect: Connection refused (error 111)
I20260926 03:11:50.454468 23304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43193 } attempt: 3
I20260926 03:11:50.455742 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.455294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43193 (local address 127.0.0.1:56442)
0926 03:11:50.455499 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:50.455589 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 03:11:50.456732 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.456394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43193 (local address 127.0.0.1:56444)
0926 03:11:50.456525 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:50.456595 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 03:11:50.456946 23304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43193: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43193: connect: Connection refused (error 111)
I20260926 03:11:50.600198 23304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43193 } attempt: 4
I20260926 03:11:50.601447 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.600917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43193 (local address 127.0.0.1:56458)
0926 03:11:50.601222 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:50.601313 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260926 03:11:50.602444 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.602082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43193 (local address 127.0.0.1:56462)
0926 03:11:50.602229 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:50.602309 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 03:11:50.602655 23304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43193: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43193: connect: Connection refused (error 111)
W20260926 03:11:50.604985 23311 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:11:50.605317 23311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:50.605404 23311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:50.605515 23311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:11:50.613037 23311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:11:50.613144 23311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:50.613216 23311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:50.613283 23311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:11:50.613361 23311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:11:50.623587 23311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35105,127.0.97.61:43193,127.0.97.60:45059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:43193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:50.624938 23311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:50.626225 23311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:50.638931 23317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:50.639004 23319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:50.639168 23316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:50.644353 23311 server_base.cc:1034] running on GCE node
I20260926 03:11:50.645283 23311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:50.646924 23311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:50.648183 23311 hybrid_clock.cc:648] HybridClock initialized: now 1790392310648106 us; error 60 us; skew 500 ppm
I20260926 03:11:50.648550 23311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:11:50 dist-test-slave-hmbw krb5kdc[23224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:11:50.651918 23311 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260926 03:11:50.653489 23311 webserver.cc:460] Webserver started at http://127.0.97.61:46471/ using document root <none> and password file <none>
I20260926 03:11:50.654237 23311 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:50.654331 23311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:50.654673 23311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:50.657933 23311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-1/data/instance:
uuid: "38c70cda3e594b348bcefa70098a5a70"
format_stamp: "Formatted at 2026-09-26 03:11:50 on dist-test-slave-hmbw"
server_key: "bb7dacb521a97dc86e164c62acddba0e"
server_key_iv: "3fa216b14404cdbdccf152f788a49ece"
server_key_version: "encryptionkey@0"
I20260926 03:11:50.658859 23311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-1/wal/instance:
uuid: "38c70cda3e594b348bcefa70098a5a70"
format_stamp: "Formatted at 2026-09-26 03:11:50 on dist-test-slave-hmbw"
server_key: "bb7dacb521a97dc86e164c62acddba0e"
server_key_iv: "3fa216b14404cdbdccf152f788a49ece"
server_key_version: "encryptionkey@0"
I20260926 03:11:50.664764 23311 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260926 03:11:50.669178 23326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:50.671339 23311 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:11:50.671545 23311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-1/wal
uuid: "38c70cda3e594b348bcefa70098a5a70"
format_stamp: "Formatted at 2026-09-26 03:11:50 on dist-test-slave-hmbw"
server_key: "bb7dacb521a97dc86e164c62acddba0e"
server_key_iv: "3fa216b14404cdbdccf152f788a49ece"
server_key_version: "encryptionkey@0"
I20260926 03:11:50.671789 23311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:50.692215 23311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:50.696787 23311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:50.697055 23311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:50.715544 23311 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:43193
I20260926 03:11:50.715564 23377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:43193 every 8 connection(s)
I20260926 03:11:50.717026 23311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-1/data/info.pb
I20260926 03:11:50.720947 23378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:50.721441 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 23311
I20260926 03:11:50.721599 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-1/wal/instance
I20260926 03:11:50.722014 388 external_mini_cluster.cc:1428] Setting key 9157869f0b8357e2443c664886f79024
I20260926 03:11:50.727736 23378 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:11:50 dist-test-slave-hmbw krb5kdc[23224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260926 03:11:50.750116 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.730580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56536 (local address 127.0.97.62:35105)
0926 03:11:50.730787 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:50.730793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:50.732175 (+ 1382us) server_negotiation.cc:407] Connection header received
0926 03:11:50.734167 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:50.734173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:50.734263 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:50.734385 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:50.736125 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:50.737123 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:50.738904 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:50.739555 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:50.739691 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:50.743776 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:50.743785 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:50.743790 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:50.743839 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:50.746794 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:50.747639 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:50.747645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:50.747649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:50.747776 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:50.748245 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:50.748249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:50.748251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:50.748558 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:50.748699 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:50.749467 (+ 768us) server_negotiation.cc:299] Negotiation successful
0926 03:11:50.749704 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:11:50.751111 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.731136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35105 (local address 127.0.0.1:56536)
0926 03:11:50.731977 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:50.732027 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:50.733861 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:50.734710 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:50.734719 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:50.735327 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:11:50.735925 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:50.735945 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:50.737354 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:50.737358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:50.738726 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:50.738743 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:50.739844 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:50.739851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:50.740106 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:50.740859 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:11:50.740884 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:11:50.743434 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:11:50.746960 (+ 3526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:50.746971 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:50.746996 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:50.747488 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:50.747903 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:50.747908 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:50.747911 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:50.748107 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:50.748688 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:11:50.748696 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:11:50.749091 (+ 395us) client_negotiation.cc:769] Sending connection context
0926 03:11:50.749344 (+ 253us) client_negotiation.cc:240] Negotiation successful
0926 03:11:50.749889 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":344,"threads_started":1}
I20260926 03:11:50.752976 23378 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43193 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:11:50 dist-test-slave-hmbw krb5kdc[23224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:11:50.769601 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.753658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43193 (local address 127.0.0.1:56472)
0926 03:11:50.753835 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:50.753852 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:50.753940 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:50.754987 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:50.754991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:50.755422 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:11:50.755574 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:50.755586 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:50.757470 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:50.757475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:50.758416 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:50.758430 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:50.759559 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:50.759565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:50.759750 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:50.760519 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:11:50.760548 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:11:50.762900 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:11:50.766923 (+ 4023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:50.766927 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:50.766930 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:50.767351 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:50.767758 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:50.767762 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:50.767764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:50.767908 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:50.768664 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:11:50.768669 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:11:50.768843 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 03:11:50.768983 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 03:11:50.769272 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 03:11:50.770010 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.753909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56472 (local address 127.0.97.61:43193)
0926 03:11:50.754491 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:50.754556 (+ 65us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:50.754583 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:11:50.754628 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:50.754634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:50.754846 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:50.754980 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:50.755873 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:50.757286 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:50.758581 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:50.759392 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:50.759521 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:50.763157 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:50.763175 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:50.763195 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:50.763235 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:50.766758 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:50.767495 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:50.767505 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:50.767509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:50.767659 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:50.768019 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:50.768024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:50.768027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:50.768497 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:50.768728 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:50.769161 (+ 433us) server_negotiation.cc:299] Negotiation successful
0926 03:11:50.769573 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":256,"threads_started":1}
I20260926 03:11:50.772666 23378 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:11:50.773808 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.773303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45059 (local address 127.0.0.1:49994)
0926 03:11:50.773479 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:50.773651 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260926 03:11:50.774049 23329 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45059: connect: Connection refused (error 111)
I20260926 03:11:50.775741 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.775310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45059 (local address 127.0.0.1:50002)
0926 03:11:50.775522 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:50.775614 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260926 03:11:50.776029 23378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45059: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45059: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260926 03:11:50.783747 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuHAswW/build/asan/bin/kudu
/tmp/dist-test-taskuHAswW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45059
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35105,127.0.97.61:43193,127.0.97.60:45059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:11:50.825363 23378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } attempt: 1
I20260926 03:11:50.826651 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.826134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45059 (local address 127.0.0.1:50018)
0926 03:11:50.826356 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:50.826442 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 03:11:50.827798 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.827366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45059 (local address 127.0.0.1:50030)
0926 03:11:50.827548 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:50.827645 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 03:11:50.828038 23378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45059: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45059: connect: Connection refused (error 111)
I20260926 03:11:50.896335 23378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } attempt: 2
I20260926 03:11:50.897574 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.897085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45059 (local address 127.0.0.1:50040)
0926 03:11:50.897320 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:50.897414 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 03:11:50.898684 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.898237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45059 (local address 127.0.0.1:50052)
0926 03:11:50.898426 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:50.898544 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 03:11:50.898934 23378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45059: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45059: connect: Connection refused (error 111)
I20260926 03:11:50.901913 23304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43193 } attempt: 5
Sep 26 03:11:50 dist-test-slave-hmbw krb5kdc[23224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:11:50.916787 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.902616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43193 (local address 127.0.0.1:56480)
0926 03:11:50.902789 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:50.902816 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:50.902937 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:50.903292 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:50.903297 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:50.903693 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:11:50.903891 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:50.903900 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:50.905029 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:50.905035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:50.906399 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:50.906418 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:50.907200 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:50.907206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:50.907384 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:50.908007 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:11:50.908031 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:11:50.910480 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:11:50.914407 (+ 3927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:50.914412 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:50.914414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:50.914846 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:50.915252 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:50.915255 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:50.915258 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:50.915418 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:50.916017 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:11:50.916020 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:11:50.916151 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 03:11:50.916255 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 03:11:50.916527 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 03:11:50.916949 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.902809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56480 (local address 127.0.97.61:43193)
0926 03:11:50.902984 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:50.902990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:50.903009 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:11:50.903067 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:50.903073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:50.903151 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:50.903232 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:50.904040 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:50.904854 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:50.906624 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:50.907040 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:50.907141 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:50.910776 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:50.910784 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:50.910789 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:50.910834 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:50.914217 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:50.914979 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:50.914985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:50.914988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:50.915143 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:50.915524 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:50.915529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:50.915532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:50.915916 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:50.916064 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:50.916402 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 03:11:50.916631 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:11:50.917938 23304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:11:50.919046 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.918535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45059 (local address 127.0.0.1:50068)
0926 03:11:50.918738 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:50.918844 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 03:11:50.920195 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.919789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45059 (local address 127.0.0.1:50078)
0926 03:11:50.919948 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:50.920059 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 03:11:50.920414 23304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45059: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45059: connect: Connection refused (error 111)
I20260926 03:11:50.971652 23304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } attempt: 1
I20260926 03:11:50.972946 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.972400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45059 (local address 127.0.0.1:50088)
0926 03:11:50.972626 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:50.972738 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 03:11:50.974028 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.973637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45059 (local address 127.0.0.1:50090)
0926 03:11:50.973792 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:50.973883 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 03:11:50.974303 23304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45059: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45059: connect: Connection refused (error 111)
I20260926 03:11:50.990188 23378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } attempt: 3
I20260926 03:11:50.991456 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.990949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45059 (local address 127.0.0.1:50098)
0926 03:11:50.991193 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:50.991297 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260926 03:11:50.992566 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:50.992128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45059 (local address 127.0.0.1:50104)
0926 03:11:50.992361 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:50.992441 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260926 03:11:50.992770 23378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45059: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45059: connect: Connection refused (error 111)
I20260926 03:11:51.042587 23304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } attempt: 2
I20260926 03:11:51.043958 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:51.043339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45059 (local address 127.0.0.1:50108)
0926 03:11:51.043613 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:51.043737 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 03:11:51.045534 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:51.045081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45059 (local address 127.0.0.1:50122)
0926 03:11:51.045253 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:51.045364 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 03:11:51.045852 23304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45059: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45059: connect: Connection refused (error 111)
I20260926 03:11:51.136051 23378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } attempt: 4
I20260926 03:11:51.137409 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:51.136813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45059 (local address 127.0.0.1:50128)
0926 03:11:51.137069 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:51.137234 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 03:11:51.139182 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:51.138642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45059 (local address 127.0.0.1:50132)
0926 03:11:51.138920 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:51.139029 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260926 03:11:51.139528 23378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45059: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45059: connect: Connection refused (error 111)
I20260926 03:11:51.152104 23304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } attempt: 3
I20260926 03:11:51.153748 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:51.153016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45059 (local address 127.0.0.1:50148)
0926 03:11:51.153370 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:51.153515 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260926 03:11:51.154464 23385 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:11:51.154954 23385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:11:51.155061 23385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:11:51.155187 23385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 03:11:51.155414 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:51.154936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45059 (local address 127.0.0.1:50164)
0926 03:11:51.155130 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:51.155235 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 03:11:51.155714 23304 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45059: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45059: connect: Connection refused (error 111)
W20260926 03:11:51.163426 23385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:11:51.163551 23385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:11:51.163607 23385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:11:51.163650 23385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:11:51.163712 23385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:11:51.175554 23385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35105,127.0.97.61:43193,127.0.97.60:45059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:11:51.177927 23385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:11:51.179623 23385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:11:51.194319 23390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:51.194941 23393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:11:51.195739 23391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:11:51.196681 23385 server_base.cc:1034] running on GCE node
I20260926 03:11:51.198215 23385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:11:51.200273 23385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:11:51.201578 23385 hybrid_clock.cc:648] HybridClock initialized: now 1790392311201510 us; error 63 us; skew 500 ppm
I20260926 03:11:51.201996 23385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:11:51 dist-test-slave-hmbw krb5kdc[23224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:11:51.206248 23385 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260926 03:11:51.208630 23385 webserver.cc:460] Webserver started at http://127.0.97.60:44767/ using document root <none> and password file <none>
I20260926 03:11:51.209664 23385 fs_manager.cc:359] Metadata directory not provided
I20260926 03:11:51.209798 23385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:11:51.210480 23385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:11:51.214308 23385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-2/data/instance:
uuid: "97f301ad48854573b28645c475d27d32"
format_stamp: "Formatted at 2026-09-26 03:11:51 on dist-test-slave-hmbw"
server_key: "49af97deb3879e2062f6d2a77272d777"
server_key_iv: "a753f0c59b21869777e76c2c7325628e"
server_key_version: "encryptionkey@0"
I20260926 03:11:51.215322 23385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-2/wal/instance:
uuid: "97f301ad48854573b28645c475d27d32"
format_stamp: "Formatted at 2026-09-26 03:11:51 on dist-test-slave-hmbw"
server_key: "49af97deb3879e2062f6d2a77272d777"
server_key_iv: "a753f0c59b21869777e76c2c7325628e"
server_key_version: "encryptionkey@0"
I20260926 03:11:51.223892 23385 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260926 03:11:51.228801 23400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:11:51.231511 23385 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260926 03:11:51.231858 23385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-2/wal
uuid: "97f301ad48854573b28645c475d27d32"
format_stamp: "Formatted at 2026-09-26 03:11:51 on dist-test-slave-hmbw"
server_key: "49af97deb3879e2062f6d2a77272d777"
server_key_iv: "a753f0c59b21869777e76c2c7325628e"
server_key_version: "encryptionkey@0"
I20260926 03:11:51.232187 23385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:11:51.253422 23385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:11:51.260594 23385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:11:51.260978 23385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:11:51.286007 23385 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:45059
I20260926 03:11:51.286031 23451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:45059 every 8 connection(s)
I20260926 03:11:51.287676 23385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-2/data/info.pb
I20260926 03:11:51.291143 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuHAswW/build/asan/bin/kudu as pid 23385
I20260926 03:11:51.291376 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuHAswW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391950058021-388-0/minicluster-data/master-2/wal/instance
I20260926 03:11:51.292084 388 external_mini_cluster.cc:1428] Setting key 6385bdf499adb40a48dcf88d5858fd5d
I20260926 03:11:51.293473 23452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:11:51.305120 23452 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:11:51 dist-test-slave-hmbw krb5kdc[23224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 03:11:51 dist-test-slave-hmbw krb5kdc[23224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:11:51.323825 23455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:51.294810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50178 (local address 127.0.97.60:45059)
0926 03:11:51.296004 (+ 1194us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:51.296024 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:51.296529 (+ 505us) server_negotiation.cc:407] Connection header received
0926 03:11:51.299702 (+ 3173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:51.299739 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:51.300329 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:51.300956 (+ 627us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:51.302984 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.304535 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.306050 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.306934 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.307133 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:51.312832 (+ 5699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:51.312844 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:51.312863 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:51.312937 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:51.318070 (+ 5133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:51.319550 (+ 1480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:51.319564 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:51.319573 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:51.319768 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:51.320402 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:51.320407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:51.320409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:51.321254 (+ 845us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:51.321521 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:51.322113 (+ 592us) server_negotiation.cc:299] Negotiation successful
0926 03:11:51.322610 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":480,"threads_started":1}
I20260926 03:11:51.327651 23456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:51.308157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56544 (local address 127.0.97.62:35105)
0926 03:11:51.308919 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:51.308925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:51.309346 (+ 421us) server_negotiation.cc:407] Connection header received
0926 03:11:51.309572 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:51.309576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:51.309827 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:51.310059 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:51.311808 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.312670 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.314634 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.315081 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.315212 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:51.319687 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:51.319695 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:51.319699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:51.319741 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:51.323100 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:51.323877 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:51.323881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:51.323884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:51.324069 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:51.325123 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:51.325131 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:51.325135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:51.326097 (+ 962us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:51.326464 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:51.327094 (+ 630us) server_negotiation.cc:299] Negotiation successful
0926 03:11:51.327328 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":415,"threads_started":1}
I20260926 03:11:51.327744 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:51.308159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35105 (local address 127.0.0.1:56544)
0926 03:11:51.309178 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:51.309227 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:51.309420 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:51.310262 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:51.310273 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:51.311228 (+ 955us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:11:51.311583 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:51.311599 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.312916 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.312921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:51.314413 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:51.314433 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.315313 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.315320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:51.315511 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:51.316125 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:11:51.316148 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:11:51.319168 (+ 3020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:11:51.323265 (+ 4097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:51.323278 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:51.323288 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:51.323725 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:51.324460 (+ 735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:51.324466 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:51.324469 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:51.324638 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:51.326429 (+ 1791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:11:51.326436 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:11:51.326717 (+ 281us) client_negotiation.cc:769] Sending connection context
0926 03:11:51.326892 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 03:11:51.327286 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"thread_start_us":626,"threads_started":1}
I20260926 03:11:51.329595 23452 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43193 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:11:51.332965 23304 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } attempt: 4
Sep 26 03:11:51 dist-test-slave-hmbw krb5kdc[23224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 26 03:11:51 dist-test-slave-hmbw krb5kdc[23224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 26 03:11:51 dist-test-slave-hmbw krb5kdc[23224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 03:11:51.348681 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:51.330995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43193 (local address 127.0.0.1:56494)
0926 03:11:51.331245 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:51.331268 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:51.331413 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:51.331977 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:51.331984 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:51.332912 (+ 928us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:11:51.333127 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:51.333141 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.334789 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.334794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:51.335958 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:51.335975 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.336956 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.336961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:51.337153 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:51.337868 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:11:51.337896 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:11:51.340567 (+ 2671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:11:51.345248 (+ 4681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:51.345254 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:51.345257 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:51.345927 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:51.346442 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:51.346447 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:51.346451 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:51.346723 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:51.347552 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:11:51.347556 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:11:51.347707 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 03:11:51.347866 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 03:11:51.348213 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 03:11:51.349077 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:51.331115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56494 (local address 127.0.97.61:43193)
0926 03:11:51.331336 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:51.331341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:51.331361 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:11:51.331691 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:51.331700 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:51.331788 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:51.331899 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:51.333574 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.334615 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.336151 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.336805 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.336935 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:51.340848 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:51.340855 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:51.340858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:51.340899 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:51.345020 (+ 4121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:51.346092 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:51.346099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:51.346104 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:51.346287 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:51.346872 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:51.346878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:51.346882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:51.347286 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:51.347472 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:51.348039 (+ 567us) server_negotiation.cc:299] Negotiation successful
0926 03:11:51.348580 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 03:11:51.350363 23456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:51.329529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56554 (local address 127.0.97.62:35105)
0926 03:11:51.329691 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:51.329696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:51.330184 (+ 488us) server_negotiation.cc:407] Connection header received
0926 03:11:51.330274 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:51.330279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:51.330390 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:51.330489 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:51.332699 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.333953 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.335905 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.336465 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.336687 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:51.343279 (+ 6592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:51.343286 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:51.343292 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:51.343337 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:51.346537 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:51.347468 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:51.347474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:51.347477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:51.347642 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:51.348212 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:51.348217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:51.348219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:51.348642 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:51.348937 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:51.349808 (+ 871us) server_negotiation.cc:299] Negotiation successful
0926 03:11:51.350095 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 03:11:51.351012 23452 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:11:51.354964 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:51.333921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45059 (local address 127.0.0.1:50186)
0926 03:11:51.334239 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:51.334261 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:51.334373 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:51.334921 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:51.334926 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:51.335451 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:11:51.335666 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:51.335678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.336957 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.336962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:51.338819 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:51.338859 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.340244 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.340252 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:51.340494 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:51.342256 (+ 1762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:11:51.342283 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:11:51.345086 (+ 2803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:11:51.349641 (+ 4555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:51.349649 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:51.349653 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:51.350338 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:51.351572 (+ 1234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:51.351577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:51.351580 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:51.351837 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:51.352849 (+ 1012us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:11:51.352856 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:11:51.353249 (+ 393us) client_negotiation.cc:769] Sending connection context
0926 03:11:51.353734 (+ 485us) client_negotiation.cc:240] Negotiation successful
0926 03:11:51.354363 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260926 03:11:51.355543 23455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:51.333999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50186 (local address 127.0.97.60:45059)
0926 03:11:51.334601 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:51.334606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:51.334620 (+ 14us) server_negotiation.cc:407] Connection header received
0926 03:11:51.334666 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:51.334669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:51.334758 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:51.334865 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:51.335916 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.336792 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.339300 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.340010 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.340185 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:51.345311 (+ 5126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:51.345317 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:51.345322 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:51.345378 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:51.349338 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:51.351137 (+ 1799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:51.351144 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:51.351148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:51.351384 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:51.352021 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:51.352025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:51.352028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:51.352604 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:51.352831 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:51.354534 (+ 1703us) server_negotiation.cc:299] Negotiation successful
0926 03:11:51.354864 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448}
Sep 26 03:11:51 dist-test-slave-hmbw krb5kdc[23224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 03:11:51.363147 23304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7f16928ee304c48ab7c7282b066c89a: Bootstrap starting.
Sep 26 03:11:51 dist-test-slave-hmbw krb5kdc[23224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:11:51.367607 23304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7f16928ee304c48ab7c7282b066c89a: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:51.369087 23304 log.cc:824] T 00000000000000000000000000000000 P e7f16928ee304c48ab7c7282b066c89a: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:51.370210 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:51.352161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45059 (local address 127.0.0.1:50196)
0926 03:11:51.352396 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:51.352432 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:51.352556 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:51.353541 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:51.353546 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:51.353997 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:11:51.354178 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:51.354188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.356114 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.356118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:51.357267 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:51.357283 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.359139 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.359144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:51.359348 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:51.360198 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:11:51.360222 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:11:51.363173 (+ 2951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:11:51.367020 (+ 3847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:51.367025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:51.367029 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:51.367546 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:51.368076 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:51.368081 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:51.368086 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:51.368253 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:51.368939 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:11:51.368944 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:11:51.369108 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 03:11:51.369262 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 03:11:51.369631 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 03:11:51.372165 23458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:51.352184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50196 (local address 127.0.97.60:45059)
0926 03:11:51.352896 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:51.352903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:51.352929 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:11:51.352998 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:51.353003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:51.353103 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:51.353224 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:51.354371 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.355986 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.357433 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.358397 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.358887 (+ 490us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:51.363449 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:51.363456 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:51.363459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:51.363494 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:51.366837 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:51.367706 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:51.367712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:51.367716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:51.367921 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:51.368378 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:51.368383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:51.368386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:51.368779 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:51.368985 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:51.371210 (+ 2225us) server_negotiation.cc:299] Negotiation successful
0926 03:11:51.371650 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":1027,"threads_started":1}
I20260926 03:11:51.373725 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:51.353317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56496 (local address 127.0.97.61:43193)
0926 03:11:51.353782 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:51.353789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:51.353812 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:11:51.353887 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:51.353893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:51.354183 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:51.354323 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:51.356692 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.357727 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.359071 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.359693 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.359840 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:51.365025 (+ 5185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:51.365033 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:51.365038 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:51.365089 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:51.368553 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:51.369576 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:51.369583 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:51.369586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:51.369999 (+ 413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:51.370863 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:51.370871 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:51.370874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:51.371774 (+ 900us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:51.372195 (+ 421us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:51.372960 (+ 765us) server_negotiation.cc:299] Negotiation successful
0926 03:11:51.373346 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201}
I20260926 03:11:51.374038 23304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7f16928ee304c48ab7c7282b066c89a: No bootstrap required, opened a new log
I20260926 03:11:51.378377 23452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97f301ad48854573b28645c475d27d32: Bootstrap starting.
I20260926 03:11:51.380288 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:11:51.382833 23279 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e7f16928ee304c48ab7c7282b066c89a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56554
I20260926 03:11:51.383701 23452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97f301ad48854573b28645c475d27d32: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:51.385821 23452 log.cc:824] T 00000000000000000000000000000000 P 97f301ad48854573b28645c475d27d32: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:51.386852 23304 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7f16928ee304c48ab7c7282b066c89a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f16928ee304c48ab7c7282b066c89a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "38c70cda3e594b348bcefa70098a5a70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43193 } } peers { permanent_uuid: "97f301ad48854573b28645c475d27d32" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } }
I20260926 03:11:51.389119 23304 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7f16928ee304c48ab7c7282b066c89a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f16928ee304c48ab7c7282b066c89a, State: Initialized, Role: FOLLOWER
I20260926 03:11:51.390003 23304 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7f16928ee304c48ab7c7282b066c89a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f16928ee304c48ab7c7282b066c89a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "38c70cda3e594b348bcefa70098a5a70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43193 } } peers { permanent_uuid: "97f301ad48854573b28645c475d27d32" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } }
I20260926 03:11:51.391575 23452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97f301ad48854573b28645c475d27d32: No bootstrap required, opened a new log
I20260926 03:11:51.391535 23461 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7f16928ee304c48ab7c7282b066c89a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f16928ee304c48ab7c7282b066c89a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "38c70cda3e594b348bcefa70098a5a70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43193 } } peers { permanent_uuid: "97f301ad48854573b28645c475d27d32" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } } }
I20260926 03:11:51.392161 23461 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7f16928ee304c48ab7c7282b066c89a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:11:51.393313 23304 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7f16928ee304c48ab7c7282b066c89a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:11:51.393437 23279 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7f16928ee304c48ab7c7282b066c89a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 03:11:51.393702 23279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7f16928ee304c48ab7c7282b066c89a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:51.397293 23279 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7f16928ee304c48ab7c7282b066c89a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f16928ee304c48ab7c7282b066c89a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "38c70cda3e594b348bcefa70098a5a70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43193 } } peers { permanent_uuid: "97f301ad48854573b28645c475d27d32" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } }
I20260926 03:11:51.399238 23279 leader_election.cc:288] T 00000000000000000000000000000000 P e7f16928ee304c48ab7c7282b066c89a [CANDIDATE]: Term 1 election: Requested vote from peers 38c70cda3e594b348bcefa70098a5a70 (127.0.97.61:43193), 97f301ad48854573b28645c475d27d32 (127.0.97.60:45059)
I20260926 03:11:51.407353 23452 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97f301ad48854573b28645c475d27d32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f16928ee304c48ab7c7282b066c89a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "38c70cda3e594b348bcefa70098a5a70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43193 } } peers { permanent_uuid: "97f301ad48854573b28645c475d27d32" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } }
I20260926 03:11:51.408032 23452 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97f301ad48854573b28645c475d27d32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f301ad48854573b28645c475d27d32, State: Initialized, Role: FOLLOWER
I20260926 03:11:51.408787 23452 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97f301ad48854573b28645c475d27d32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f16928ee304c48ab7c7282b066c89a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "38c70cda3e594b348bcefa70098a5a70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43193 } } peers { permanent_uuid: "97f301ad48854573b28645c475d27d32" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } }
I20260926 03:11:51.411191 23472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97f301ad48854573b28645c475d27d32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f16928ee304c48ab7c7282b066c89a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "38c70cda3e594b348bcefa70098a5a70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43193 } } peers { permanent_uuid: "97f301ad48854573b28645c475d27d32" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } } }
I20260926 03:11:51.413307 23452 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97f301ad48854573b28645c475d27d32 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:11:51.413431 23427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7f16928ee304c48ab7c7282b066c89a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "97f301ad48854573b28645c475d27d32"
I20260926 03:11:51.414685 23427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97f301ad48854573b28645c475d27d32 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:11:51.420930 23427 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 97f301ad48854573b28645c475d27d32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f16928ee304c48ab7c7282b066c89a in term 1.
I20260926 03:11:51.422406 23255 leader_election.cc:302] T 00000000000000000000000000000000 P e7f16928ee304c48ab7c7282b066c89a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97f301ad48854573b28645c475d27d32, e7f16928ee304c48ab7c7282b066c89a; no voters:
I20260926 03:11:51.423183 23461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7f16928ee304c48ab7c7282b066c89a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:11:51.427135 23461 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7f16928ee304c48ab7c7282b066c89a [term 1 LEADER]: Becoming Leader. State: Replica: e7f16928ee304c48ab7c7282b066c89a, State: Running, Role: LEADER
I20260926 03:11:51.427975 23461 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7f16928ee304c48ab7c7282b066c89a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f16928ee304c48ab7c7282b066c89a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "38c70cda3e594b348bcefa70098a5a70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43193 } } peers { permanent_uuid: "97f301ad48854573b28645c475d27d32" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } }
I20260926 03:11:51.412952 23472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97f301ad48854573b28645c475d27d32 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 03:11:51.432385 23258 tablet.cc:2369] T 00000000000000000000000000000000 P e7f16928ee304c48ab7c7282b066c89a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:11:51.436158 23481 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 03:11:51.438748 23378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } attempt: 5
I20260926 03:11:51.435910 23478 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7f16928ee304c48ab7c7282b066c89a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7f16928ee304c48ab7c7282b066c89a. Latest consensus state: current_term: 1 leader_uuid: "e7f16928ee304c48ab7c7282b066c89a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f16928ee304c48ab7c7282b066c89a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "38c70cda3e594b348bcefa70098a5a70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43193 } } peers { permanent_uuid: "97f301ad48854573b28645c475d27d32" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } } }
I20260926 03:11:51.441214 23478 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7f16928ee304c48ab7c7282b066c89a [sys.catalog]: This master's current role is: LEADER
Sep 26 03:11:51 dist-test-slave-hmbw krb5kdc[23224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790392310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 03:11:51.460916 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:51.439553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45059 (local address 127.0.0.1:50210)
0926 03:11:51.439919 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0926 03:11:51.439938 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:11:51.440054 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:11:51.441266 (+ 1212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:11:51.441272 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:11:51.441913 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:11:51.442181 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:51.442196 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.444066 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.444073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:51.445727 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:11:51.445752 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.447410 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.447439 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:11:51.447801 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:51.448770 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:11:51.448819 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:11:51.453205 (+ 4386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:11:51.457806 (+ 4601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:51.457812 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:51.457816 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:51.458431 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:51.458926 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:11:51.458932 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:11:51.458936 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:11:51.459114 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:11:51.459857 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:11:51.459863 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:11:51.460027 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 03:11:51.460190 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 03:11:51.460522 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236}
I20260926 03:11:51.462095 23458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:51.440416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50210 (local address 127.0.97.60:45059)
0926 03:11:51.440644 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:51.440651 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:51.440671 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:11:51.440739 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:51.440744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:51.440888 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:51.440997 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:51.442465 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.443804 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.446060 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.446891 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.447040 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:51.453574 (+ 6534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:51.453584 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:51.453590 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:51.453644 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:51.457584 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:51.458617 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:51.458623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:51.458627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:51.458792 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:51.459235 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:51.459239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:51.459242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:51.459638 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:51.459776 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:51.461493 (+ 1717us) server_negotiation.cc:299] Negotiation successful
0926 03:11:51.461757 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:11:51.443521 23237 master_runner.cc:186] Error getting master registration for 127.0.97.61:43193: OK, instance_id { permanent_uuid: "38c70cda3e594b348bcefa70098a5a70" instance_seqno: 1790392310710155 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:11:51.467235 23237 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 03:11:51.467718 23484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:11:51.472218 23378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38c70cda3e594b348bcefa70098a5a70: Bootstrap starting.
I20260926 03:11:51.477288 23378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38c70cda3e594b348bcefa70098a5a70: Neither blocks nor log segments found. Creating new log.
I20260926 03:11:51.478914 23378 log.cc:824] T 00000000000000000000000000000000 P 38c70cda3e594b348bcefa70098a5a70: Log is configured to *not* fsync() on all Append() calls
I20260926 03:11:51.485558 23378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38c70cda3e594b348bcefa70098a5a70: No bootstrap required, opened a new log
I20260926 03:11:51.489892 23456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:51.437392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56560 (local address 127.0.97.62:35105)
0926 03:11:51.437619 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:51.437625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:51.439432 (+ 1807us) server_negotiation.cc:407] Connection header received
0926 03:11:51.439979 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:51.439987 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:51.440091 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:51.440174 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:51.456431 (+ 16257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.457724 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.461413 (+ 3689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.462163 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.462273 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:51.473718 (+ 11445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:51.473727 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:51.473732 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:51.473794 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:51.478149 (+ 4355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:51.481693 (+ 3544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:51.481701 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:51.481705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:51.481939 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:51.484261 (+ 2322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:51.484269 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:51.484273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:51.484772 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:51.485131 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:51.488493 (+ 3362us) server_negotiation.cc:299] Negotiation successful
0926 03:11:51.489066 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 03:11:51.499153 23378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38c70cda3e594b348bcefa70098a5a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f16928ee304c48ab7c7282b066c89a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "38c70cda3e594b348bcefa70098a5a70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43193 } } peers { permanent_uuid: "97f301ad48854573b28645c475d27d32" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } }
I20260926 03:11:51.499780 23378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38c70cda3e594b348bcefa70098a5a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38c70cda3e594b348bcefa70098a5a70, State: Initialized, Role: FOLLOWER
I20260926 03:11:51.500675 23378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38c70cda3e594b348bcefa70098a5a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f16928ee304c48ab7c7282b066c89a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "38c70cda3e594b348bcefa70098a5a70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43193 } } peers { permanent_uuid: "97f301ad48854573b28645c475d27d32" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } }
I20260926 03:11:51.508066 23491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38c70cda3e594b348bcefa70098a5a70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f16928ee304c48ab7c7282b066c89a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "38c70cda3e594b348bcefa70098a5a70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43193 } } peers { permanent_uuid: "97f301ad48854573b28645c475d27d32" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } } }
I20260926 03:11:51.508651 23491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38c70cda3e594b348bcefa70098a5a70 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:11:51.511096 23378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38c70cda3e594b348bcefa70098a5a70 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:11:51.512876 23353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7f16928ee304c48ab7c7282b066c89a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "38c70cda3e594b348bcefa70098a5a70"
I20260926 03:11:51.513494 23353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38c70cda3e594b348bcefa70098a5a70 [term 0 FOLLOWER]: Advancing to term 1
W20260926 03:11:51.514071 23406 tablet.cc:2369] T 00000000000000000000000000000000 P 97f301ad48854573b28645c475d27d32: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:11:51.517539 23353 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 38c70cda3e594b348bcefa70098a5a70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f16928ee304c48ab7c7282b066c89a in term 1.
I20260926 03:11:51.532747 23484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:11:51.542747 23501 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 03:11:51.551671 23427 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 97f301ad48854573b28645c475d27d32 [term 1 FOLLOWER]: Refusing update from remote peer e7f16928ee304c48ab7c7282b066c89a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:11:51.553426 23478 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e7f16928ee304c48ab7c7282b066c89a [LEADER]: Connected to new peer: Peer: permanent_uuid: "97f301ad48854573b28645c475d27d32" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:11:51.554477 23353 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 38c70cda3e594b348bcefa70098a5a70 [term 1 FOLLOWER]: Refusing update from remote peer e7f16928ee304c48ab7c7282b066c89a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:11:51.560897 23455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:51.455276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50212 (local address 127.0.97.60:45059)
0926 03:11:51.455617 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:51.455625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:51.461604 (+ 5979us) server_negotiation.cc:407] Connection header received
0926 03:11:51.462027 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:51.462033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:51.462115 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:51.462185 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:51.470289 (+ 8104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.471879 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.490748 (+ 18869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.491593 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.491725 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:51.521321 (+ 29596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:51.521330 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:51.521336 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:51.521396 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:51.525585 (+ 4189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:51.532421 (+ 6836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:51.532429 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:51.532433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:51.532643 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:51.553022 (+ 20379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:51.553029 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:51.553033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:51.553484 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:51.560046 (+ 6562us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:51.560235 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 03:11:51.560526 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260926 03:11:51.562068 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:51.438020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56508 (local address 127.0.97.61:43193)
0926 03:11:51.438338 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:51.438348 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:51.454668 (+ 16320us) server_negotiation.cc:407] Connection header received
0926 03:11:51.457819 (+ 3151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:51.457825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:51.457908 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:51.457979 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:51.462959 (+ 4980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.464051 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.480140 (+ 16089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:51.480905 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:51.481023 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:51.507684 (+ 26661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:11:51.507697 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:11:51.507704 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:11:51.507766 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:11:51.512562 (+ 4796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:51.526596 (+ 14034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:51.526605 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:51.526608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:51.526959 (+ 351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:11:51.528771 (+ 1812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:11:51.528803 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:11:51.528816 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:11:51.529608 (+ 792us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:11:51.530224 (+ 616us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:51.542867 (+ 12643us) server_negotiation.cc:299] Negotiation successful
0926 03:11:51.543214 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 03:11:51.562434 23478 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e7f16928ee304c48ab7c7282b066c89a [LEADER]: Connected to new peer: Peer: permanent_uuid: "38c70cda3e594b348bcefa70098a5a70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:11:51.550518 23385 master_runner.cc:186] Error getting master registration for 127.0.97.61:43193: OK, instance_id { permanent_uuid: "38c70cda3e594b348bcefa70098a5a70" instance_seqno: 1790392310710155 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:11:51.568529 23385 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 03:11:51.582949 23491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38c70cda3e594b348bcefa70098a5a70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7f16928ee304c48ab7c7282b066c89a. Latest consensus state: current_term: 1 leader_uuid: "e7f16928ee304c48ab7c7282b066c89a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f16928ee304c48ab7c7282b066c89a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "38c70cda3e594b348bcefa70098a5a70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43193 } } peers { permanent_uuid: "97f301ad48854573b28645c475d27d32" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } } }
I20260926 03:11:51.583302 23491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38c70cda3e594b348bcefa70098a5a70 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:11:51.584159 23502 mvcc.cc:205] Tried to move back new op lower bound from 7333446908098584576 to 7333446907620429824. Current Snapshot: MvccSnapshot[applied={T|T < 7333446908098584576}]
I20260926 03:11:51.592820 23509 mvcc.cc:205] Tried to move back new op lower bound from 7333446908098584576 to 7333446907620429824. Current Snapshot: MvccSnapshot[applied={T|T < 7333446908098584576}]
I20260926 03:11:51.594769 23491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38c70cda3e594b348bcefa70098a5a70 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7f16928ee304c48ab7c7282b066c89a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f16928ee304c48ab7c7282b066c89a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "38c70cda3e594b348bcefa70098a5a70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43193 } } peers { permanent_uuid: "97f301ad48854573b28645c475d27d32" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } } }
I20260926 03:11:51.595053 23491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38c70cda3e594b348bcefa70098a5a70 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:11:51.599344 23478 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7f16928ee304c48ab7c7282b066c89a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7f16928ee304c48ab7c7282b066c89a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f16928ee304c48ab7c7282b066c89a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "38c70cda3e594b348bcefa70098a5a70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43193 } } peers { permanent_uuid: "97f301ad48854573b28645c475d27d32" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } } }
I20260926 03:11:51.599643 23478 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7f16928ee304c48ab7c7282b066c89a [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:51.600700 23472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97f301ad48854573b28645c475d27d32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7f16928ee304c48ab7c7282b066c89a. Latest consensus state: current_term: 1 leader_uuid: "e7f16928ee304c48ab7c7282b066c89a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f16928ee304c48ab7c7282b066c89a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "38c70cda3e594b348bcefa70098a5a70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43193 } } peers { permanent_uuid: "97f301ad48854573b28645c475d27d32" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } } }
I20260926 03:11:51.601034 23472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97f301ad48854573b28645c475d27d32 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:11:51.602839 23484 catalog_manager.cc:1347] Generated new cluster ID: 23355f11e52d4b55b16bb6fdb94e7a4e
I20260926 03:11:51.602957 23484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:11:51.603142 23461 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7f16928ee304c48ab7c7282b066c89a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7f16928ee304c48ab7c7282b066c89a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f16928ee304c48ab7c7282b066c89a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "38c70cda3e594b348bcefa70098a5a70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43193 } } peers { permanent_uuid: "97f301ad48854573b28645c475d27d32" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } } }
I20260926 03:11:51.603528 23461 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7f16928ee304c48ab7c7282b066c89a [sys.catalog]: This master's current role is: LEADER
I20260926 03:11:51.606928 23510 mvcc.cc:205] Tried to move back new op lower bound from 7333446908098584576 to 7333446907620429824. Current Snapshot: MvccSnapshot[applied={T|T < 7333446908098584576}]
I20260926 03:11:51.608832 23514 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 03:11:51.610219 23472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97f301ad48854573b28645c475d27d32 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7f16928ee304c48ab7c7282b066c89a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f16928ee304c48ab7c7282b066c89a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35105 } } peers { permanent_uuid: "38c70cda3e594b348bcefa70098a5a70" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43193 } } peers { permanent_uuid: "97f301ad48854573b28645c475d27d32" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45059 } } }
I20260926 03:11:51.610461 23472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97f301ad48854573b28645c475d27d32 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:11:51.622759 23484 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:11:51.624696 23484 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:11:51.636564 23484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7f16928ee304c48ab7c7282b066c89a: Generated new TSK 0
I20260926 03:11:51.637840 23484 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 03:11:51.708058 23456 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:11:51.705066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56568 (local address 127.0.97.62:35105)
0926 03:11:51.705327 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:51.705334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:51.705683 (+ 349us) server_negotiation.cc:407] Connection header received
0926 03:11:51.705965 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:51.705971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:51.706057 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:51.706133 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:51.707805 (+ 1672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56568: BlockingRecv error: recv got EOF from 127.0.0.1:56568 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260926 03:11:51.708220 23382 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:11:51.705881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56512 (local address 127.0.97.61:43193)
0926 03:11:51.706410 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:51.706416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:51.706434 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:11:51.706651 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:51.706657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:51.706736 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:51.706824 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:51.707953 (+ 1129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56512: BlockingRecv error: recv got EOF from 127.0.0.1:56512 (error 108)
Metrics: {"server-negotiator.queue_time_us":373}
W20260926 03:11:51.710323 23455 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:11:51.706842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50216 (local address 127.0.97.60:45059)
0926 03:11:51.707022 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:51.707027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:51.708699 (+ 1672us) server_negotiation.cc:407] Connection header received
0926 03:11:51.708938 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:51.708944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:51.709026 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:51.709094 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:51.710104 (+ 1010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50216: BlockingRecv error: recv got EOF from 127.0.0.1:50216 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 03:11:51.715806 23382 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:11:51.713886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56516 (local address 127.0.97.61:43193)
0926 03:11:51.714079 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:51.714085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:51.714330 (+ 245us) server_negotiation.cc:407] Connection header received
0926 03:11:51.714579 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:51.714585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:51.714718 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:51.714819 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:51.715568 (+ 749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56516: BlockingRecv error: recv got EOF from 127.0.0.1:56516 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 03:11:51.725561 23382 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:11:51.720111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56528 (local address 127.0.97.61:43193)
0926 03:11:51.720309 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:51.720316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:51.723104 (+ 2788us) server_negotiation.cc:407] Connection header received
0926 03:11:51.723298 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:51.723304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:51.723385 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:51.723468 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:51.725329 (+ 1861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56528: BlockingRecv error: recv got EOF from 127.0.0.1:56528 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 03:11:51.740510 23382 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:11:51.736509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56534 (local address 127.0.97.61:43193)
0926 03:11:51.736725 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:51.736733 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:51.739123 (+ 2390us) server_negotiation.cc:407] Connection header received
0926 03:11:51.739343 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:51.739348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:51.739424 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:51.739509 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:51.740267 (+ 758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56534: BlockingRecv error: recv got EOF from 127.0.0.1:56534 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 03:11:51.750186 23382 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:11:51.748328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56542 (local address 127.0.97.61:43193)
0926 03:11:51.748575 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:51.748581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:51.748817 (+ 236us) server_negotiation.cc:407] Connection header received
0926 03:11:51.749031 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:51.749038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:51.749119 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:51.749203 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:51.749946 (+ 743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56542: BlockingRecv error: recv got EOF from 127.0.0.1:56542 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260926 03:11:51.769904 23382 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:11:51.767945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56550 (local address 127.0.97.61:43193)
0926 03:11:51.768147 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:51.768154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:51.768376 (+ 222us) server_negotiation.cc:407] Connection header received
0926 03:11:51.768537 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:51.768543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:51.768634 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:51.768757 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:51.769642 (+ 885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56550: BlockingRecv error: recv got EOF from 127.0.0.1:56550 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 03:11:51.791787 23382 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:11:51.789939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56562 (local address 127.0.97.61:43193)
0926 03:11:51.790139 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:51.790145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:51.790397 (+ 252us) server_negotiation.cc:407] Connection header received
0926 03:11:51.790600 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:51.790607 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:51.790714 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:51.790850 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:51.791539 (+ 689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56562: BlockingRecv error: recv got EOF from 127.0.0.1:56562 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 03:11:51.829098 23382 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:11:51.827060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56574 (local address 127.0.97.61:43193)
0926 03:11:51.827312 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:51.827320 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:51.827539 (+ 219us) server_negotiation.cc:407] Connection header received
0926 03:11:51.827704 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:51.827711 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:51.827822 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:51.827952 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:51.828804 (+ 852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56574: BlockingRecv error: recv got EOF from 127.0.0.1:56574 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260926 03:11:51.898339 23382 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:11:51.896482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56578 (local address 127.0.97.61:43193)
0926 03:11:51.896682 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:51.896688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:51.896959 (+ 271us) server_negotiation.cc:407] Connection header received
0926 03:11:51.897110 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:51.897116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:51.897216 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:51.897343 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:51.898109 (+ 766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56578: BlockingRecv error: recv got EOF from 127.0.0.1:56578 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 03:11:52.034472 23382 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:11:52.032202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56586 (local address 127.0.97.61:43193)
0926 03:11:52.032597 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:52.032607 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:52.032880 (+ 273us) server_negotiation.cc:407] Connection header received
0926 03:11:52.033036 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:52.033040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:52.033128 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:52.033252 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:52.034148 (+ 896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56586: BlockingRecv error: recv got EOF from 127.0.0.1:56586 (error 108)
Metrics: {"server-negotiator.queue_time_us":152}
W20260926 03:11:52.296902 23382 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:11:52.294704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56598 (local address 127.0.97.61:43193)
0926 03:11:52.294922 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:52.294928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:52.295239 (+ 311us) server_negotiation.cc:407] Connection header received
0926 03:11:52.295394 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:52.295400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:52.295496 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:52.295611 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:11:52.296643 (+ 1032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56598: BlockingRecv error: recv got EOF from 127.0.0.1:56598 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:11:52.546998 23501 catalog_manager.cc:1259] Loaded cluster ID: 23355f11e52d4b55b16bb6fdb94e7a4e
I20260926 03:11:52.547122 23501 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 97f301ad48854573b28645c475d27d32: loading cluster ID for follower catalog manager: success
I20260926 03:11:52.549701 23501 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 97f301ad48854573b28645c475d27d32: acquiring CA information for follower catalog manager: success
I20260926 03:11:52.551656 23501 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 97f301ad48854573b28645c475d27d32: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:11:52.614470 23514 catalog_manager.cc:1259] Loaded cluster ID: 23355f11e52d4b55b16bb6fdb94e7a4e
I20260926 03:11:52.614629 23514 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 38c70cda3e594b348bcefa70098a5a70: loading cluster ID for follower catalog manager: success
I20260926 03:11:52.619256 23514 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 38c70cda3e594b348bcefa70098a5a70: acquiring CA information for follower catalog manager: success
I20260926 03:11:52.622558 23514 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 38c70cda3e594b348bcefa70098a5a70: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:11:52.835419 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:52.813346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56602 (local address 127.0.97.61:43193)
0926 03:11:52.826673 (+ 13327us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:52.826689 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:52.827466 (+ 777us) server_negotiation.cc:407] Connection header received
0926 03:11:52.827698 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:52.827705 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:52.827934 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:52.828105 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:11:52.828701 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:52.829989 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:52.832078 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:52.832804 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:52.832956 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:52.833702 (+ 746us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:11:52.834621 (+ 919us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:11:52.834817 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:52.834927 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 03:11:52.835044 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13130,"thread_start_us":13355,"threads_started":1}
I20260926 03:11:52.863468 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:52.844567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50222 (local address 127.0.97.60:45059)
0926 03:11:52.855986 (+ 11419us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:52.855993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:52.856015 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:11:52.856146 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:52.856151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:52.856368 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:52.856580 (+ 212us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:11:52.857144 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:52.858442 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:52.860410 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:52.861085 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:52.861226 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:52.861887 (+ 661us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:11:52.862666 (+ 779us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:11:52.862820 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:52.862963 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 03:11:52.863058 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11253,"thread_start_us":11057,"threads_started":1}
I20260926 03:11:52.894107 23596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:52.870885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56576 (local address 127.0.97.62:35105)
0926 03:11:52.883402 (+ 12517us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:52.883409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:52.885349 (+ 1940us) server_negotiation.cc:407] Connection header received
0926 03:11:52.885828 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:52.885834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:52.886062 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:52.886220 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:11:52.886975 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:52.888047 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:52.890377 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:52.891019 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:52.891128 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:52.891951 (+ 823us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:11:52.892838 (+ 887us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:11:52.892921 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:52.893640 (+ 719us) server_negotiation.cc:299] Negotiation successful
0926 03:11:52.893732 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12360,"thread_start_us":12533,"threads_started":1}
I20260926 03:11:52.901319 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:52.892366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50224 (local address 127.0.97.60:45059)
0926 03:11:52.892560 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:52.892566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:52.893542 (+ 976us) server_negotiation.cc:407] Connection header received
0926 03:11:52.894914 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:52.894920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:52.895040 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:52.895107 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:11:52.895767 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:52.896871 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:52.898964 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:52.899656 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:52.899755 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:52.900435 (+ 680us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:11:52.900577 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:11:52.900658 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:52.900901 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 03:11:52.900997 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 03:11:52.902568 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:11:52.884379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56610 (local address 127.0.97.61:43193)
0926 03:11:52.884582 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 03:11:52.884588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:11:52.891676 (+ 7088us) server_negotiation.cc:407] Connection header received
0926 03:11:52.892892 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:11:52.892898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:11:52.892992 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:11:52.893051 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:11:52.894280 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:52.895457 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:52.899416 (+ 3959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:11:52.900096 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:11:52.900191 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:11:52.901349 (+ 1158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:11:52.901505 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:11:52.901574 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 03:11:52.901965 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 03:11:52.902055 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:11:52.908336 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 23237
I20260926 03:11:52.909698 23299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:11:53.066291 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 23237
W20260926 03:11:53.083598 23328 connection.cc:535] server connection from 127.0.0.1:56480 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 03:11:53.083909 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 23311
I20260926 03:11:53.084986 23373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:11:53.190876 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 23311
I20260926 03:11:53.208789 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskuHAswW/build/asan/bin/kudu pid 23385
I20260926 03:11:53.209837 23447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:11:53.316993 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuHAswW/build/asan/bin/kudu with pid 23385
2026-09-26T03:11:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4846 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9245 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (363215 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST